Apparatus for controlling contents of a computer-generated image using 3d measurements

ABSTRACT

A method for controlling a computer display cursor in an interaction region includes projecting an image of the computer display to create the interaction region. A distance is established between a first point and a second point. The first point has a predetermined relation to the projection device, and the second point has a predetermined relation to the interaction region. At least one of an orientation and a position of a pointing line is measured. The pointing line has a predetermined relation to a pointing device. The established distance and the at least one of measured position and orientation are used to control the cursor position on the interaction region.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/014,879, filed Feb. 3, 2016, which is a continuation of U.S.application Ser. No. 11/483,170 filed Jul. 7, 2006, now U.S. Pat. No.9,285,897, which claims priority to U.S. Provisional Application No.60/698,702 filed Jul. 13, 2005, and U.S. Provisional Application No.60/742,393 filed Dec. 5, 2005, all of which are hereby incorporated byreference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The invention relates generally to devices for making presentations infront of audiences and, more specifically, to devices and methods formaking presentations for which interaction with the displayedinformation. through direct-pointing is desired.

2. Background Art

Technology for presenting computer-generated images on large screens hasdeveloped to the point where such presentations are commonplace. Forexample, the software package POWERPOINT, sold by Microsoft Corp.,Redmond, Wash., may be used in combination with a so-called ‘beamer’ togenerate interactive presentation material and project for viewing by anaudience. Often, such presentations are held in rooms not specificallyequipped for the purpose of showing presentations, in which case use ismade of a portable beamer in combination with a so-called ‘laptop’computer. Under these circumstances the projection surface may be a wallof the room.

During a presentation it is desirable for the presenter to be able tomove freely in front of the audience while retaining the capability tointeract with the presentation and point to specific features on thedisplayed images. In general, interaction with a computer is oftenfacilitated by pointing devices such as a ‘mouse’ or a ‘trackball’ thatenable manipulation of a so-called ‘cursor’. Traditionally, thesedevices were physically connected to the computer, thus constraining thefreedom-of-movement of the user. More recently, however, this constrainthas been removed by the introduction of wireless pointing devices suchas the GYROMOUSE, as manufactured by Gyration, Inc.

Broadly speaking, pointing devices may be classified in two categories:a) devices for so-called ‘direct-pointing’ and b) devices for so-called‘indirect-pointing’. Direct pointing devices are those for which thephysical point-of-aim coincides with the item being pointed at, i.e., itlies on the line-of-sight. Direct pointing devices include the so-called‘laser pointer’ and the human pointing finger. Indirect pointing devicesinclude systems where the object of pointing (e.g., a cursor) bears anindirect relationship to the physical point-of-aim of the pointingdevice; examples include a mouse and a trackball. It needs no argumentthat direct-pointing systems are more natural to humans, allowing fasterand more accurate pointing actions.

Indirect pointing devices known in the art include the following. U.S.Pat. No. 4,654,648 to Herrington et al. (1987), U.S. Pat. No. 5,339,095to Redford (1994), U.S. Pat. No. 5,359,348 to Pilcher et al. (1994),U.S. Pat. No. 5,469,193 to Giobbi et al. (1995), U.S. Pat. No. 5,506,605to Paley (1996), U.S. Pat. No. 5,638,092 to Eng et al. (1997), U.S. Pat.No. 5,734,371 to Kaplan (1998), U.S. Pat. No. 5,883,616 to Koizumi etal. (1999), U.S. Pat. No. 5,898,421 to Quinn (1999), U.S. Pat. No.5,963,134 to Bowers et al. (1999), U.S. Pat. No. 5,999,167 to Marsh etal. (1999), U.S. Pat. No. 6,069,594 to Barnes et al. (2000), U.S. Pat.No. 6,130,664 to Suzuki (2000), U.S. Pat. No. 6,271,831 to Escobosa etal. (2001), U.S. Pat. No. 6,342,878 to Chevassus et al. (2002), U.S.Pat. No. 6,388,656 to Chae (2002), U.S. Pat. No. 6,411,277 toShah-Nazaroff (2002), U.S. Pat. No. 6,492,981 Stork et al. (2002), U.S.Pat. No. 6,504,526 to Mauritz (2003), U.S. Pat. No. 6,545,664 to Kim(2003), U.S. Pat. No. 6,567,071 to Curran et al. (2003) and U.S. PatentApplication Publication No. 2002/0085097 to Colmenarez et al. (2002).Each of the foregoing publications discloses a system for which the 2dimensional or 3 dimensional position orientation and/or motion of anobject, such as a handheld pointing device, are measured with respect tosome reference coordinate system using appropriate means. Such meansinclude acoustic devices, electromagnetic devices, infrared devices,visible light emitting diode (LED) devices, charge coupled devices(CCD), accelerometer and gyroscopic motion detectors, etc. Although forsome of the foregoing devices the reference coordinate system may bepositioned close to the display means, no information on the actualposition of the presentation display with respect to the system is used,causing the resulting pointing action to be inherently indirect and,hence, less natural to the human operators of such systems.

Other inherently indirect-pointing systems that do not require theposition or orientation of the pointing device to be known includedevices such as disclosed in U.S. Pat. No. 5,095,302 to McLean et al.(1992) and U.S. Pat. No. 5,668,574 to Jarlance-Huang (1997). Theforegoing patents describe indirect-pointing methods that do not providethe speed and intuitiveness afforded by direct-pointing systems.

Direct pointing devices are disclosed, for example, in U.S. Pat. No.4,823,170 to Hansen (1989), which describes a direct-pointing systemcomprising a light source, a position-sensing detector located adjacentto the light source and a focusing reflector that, in one application,is parabolic shaped and is attached to the pointing device.Additionally, procedures are described to calibrate the system. In theunderstanding of current applicant, however, the physical location ofthe position-sensing detector needs to be, at least preferably, adjacentto the display means. The system disclosed in the Hansel '170 patentcannot easily be ported to a room not specifically equipped for thissystem.

U.S. Pat. No. 5,929,444 to Leichner (1999) discloses a system primarilyintended for target shooting practice, but an application as adirect-pointing cursor control apparatus may arguably be anticipated.The system includes transmitting and detecting equipment in a fixedreference base and a moveable pointing device. A calibration routine isdescribed that aligns the detecting and transmitting means while keepingthe pointing device (i.e., a gun) aimed at the center of the target. TheLeichner '444 patent does not describe methods or means that allowdetermination of a point-of-aim of a pointing device on a target ofwhich the size and/or orientation have not been predetermined.Consequently, the system disclosed in the Leichner '444 patent is notsuitable to be used as a cursor control means for projection surfacesnot specifically adapted to be used with such a system.

U.S. Pat. No. 5,952,996 to Kim et al. (1999), U.S. Pat. No. 6,184,863 toSibert et al. (2001) and US Patent Application Publication No.2002/0084980 to White et al. (2002) disclose direct-pointing systemswhere the 3 dimensional position and/or orientation of the pointingdevice is measured with respect to sources and/or detectors, thephysical position of which in relation to the display means is presumedknown. Such systems only work in rooms specifically equipped for theiruse.

U.S. Pat. No. 5,484,966 to Segen (1996), U.S. Pat. No. 6,335,723 to Woodet al. (2002) and U.S. Pat. No. 6,507,339 to Tanaka (2003) disclosemethods suitable for direct-pointing that are useful only if thepointing device is physically close to or touching the display area orvolume, for example used with so-called ‘interactive whiteboards’. Someof the foregoing patents describe appropriate pointer calibrationroutines. Such systems are not suitable for presentations where thedisplay surface is out of the presenter's physical reach.

U.S. Pat. No. 6,104,380 to Stork et al. (2000) discloses adirect-pointing system in which at least the 3 dimensional orientationof a handheld pointing device is measured by appropriate means.Additionally, a direct measurement is made of the distance between thepointing device and the displayed image. However, the system disclosedin the Stork et al. '380 patent does not include methods to ascertainthe position and orientation of the display means relative to thepointing device. In the foregoing system, these appear to be presumedknown. This is also the case for a system disclosed in U.S. Pat. No.4,768,028 to Blackie (1988), in which the orientation of ahelmet-mounted direct-pointing apparatus is measuredelectromagnetically. The foregoing systems therefore appear to beill-suited to operate in rooms not specifically equipped forpresentation purposes.

U.S. Pat. No. 6,373,961 to Richardson et al. (2002) discloses adirect-pointing system using helmet-mounted eye tracking means. Thepoint-of-gaze relative to the helmet is measured as well as the positionand orientation of the helmet relative to the display. The latter isaccomplished by equipping the helmet either with means to image sourcesmounted at fixed positions around the display or with means to image adisplayed calibration pattern. Of course, the foregoing system relies onsophisticated helmet mounted equipment capable of, among other things,tracking eye-movement. Moreover, such a system relies on an unobstructedline-of-sight with respect to the display and a substantially constantdistance from the display to the helmet-mounted equipment. The disclosedinvention does not lend itself to be easily used by a human operator inan arbitrary (not predetermined) presentation setting.

U.S. Pat. No. 6,385,331 to Harakawa et al. (2002) discloses a systemthat uses infrared technology in combination with image recognitionsoftware to distinguish pointing gestures made by the presenter, withoutthe need for an artificial pointing device. The disclosed system,however, requires the presentation room to be set up with highly tunedand sophisticated equipment, and is therefore not easily ported to adifferent venue.

U.S. Pat. No. 6,404,416 to Kahn et al. (2002) discloses adirect-pointing system where a handheld pointing device is equipped withan optical sensor. In such system either the display is required to beof a specific type (e.g., a cathode ray-based display that uses anelectron: beam) or the displayed image is required to be enhanced bytimed and specific emanations. When pointed to the display, a handheldpointing device may detect the electron beam or the timed emanations,and the timing of these detections may then be used to ascertain thepoint-of-aim. The disclosed system is somewhat similar to thetechnologies used for so-called light guns in video games as disclosed,for example, in U.S. Pat. No. 6,171,190 to Thanasack et al. (2001) andU.S. Pat. No. 6,545,661 to Goschy et al. (2003). Of course, such systemsrequire either a specific display apparatus or a specializedmodification of the displayed images. Moreover, an uncompromisedline-of-sight between the pointer and the display is a prerequisite forsuch systems.

U.S. Patent Application Publication No. 2002/0089489 to Carpenter (2002)discloses a direct-pointing system that, in one embodiment, positions acomputer cursor at a light spot projected by a laser-pointer. The systemrelies on the use of an image-capturing device to compare a capturedimage with a projected image. As such, the system disclosed in the '489application publication makes use of calibration routines in which theuser is required to highlight computer-generated calibration marks withthe laser pointer. The system disclosed in the '489 patent applicationpublication is not unlike a system disclosed in U.S. Pat. No. 5,502,459to Marshall et al. (1996). Also, U.S. Pat. No. 5,654,741 to Sampsell etal. (1997), U.S. Pat. No. 6,292,171 to Fu et al. (2001), U.S. PatentApplication Publication No. 2002/0042699 to Tanaka et al. (2002) andU.S. Patent Application Publication No. 2002/0075386 to Tanaka (2002)all disclose systems that can detect a light-spot using optical means.Such systems specifically generally require the use of computationallyexpensive image processing technologies. All of these inventions requirea projection surface with adequate diffusion properties, as well as someform of optical system with a steady and uncompromised view of thedisplay area. As such, they limit the freedom-of-movement of thepresenter and place limitations on the position and opticalcharacteristics of the necessary equipment. Also, in some of theseinventions fast and involuntary movement of the user's hand may resultin a cursor that does not move smoothly or a cursor that does notperfectly track the light spot, causing possible confusion with theuser.

U.S. Patent Application Publication No. 2005/0270494 to Banning (2005)and pages 83-87 of ISSN 1476-2986 by Ashdown (2004) describedirect-pointing methods for which the spatial characteristics of aprojected display are determined by means of a user-executed calibrationprocedure. Direct-pointing cursor control is subsequently realized bymeasuring the position and/or orientation of a pointing device andmathematically intersecting the associated pointing line with theprojected display. The calibration procedure requires the user to directa sufficient number of pointing lines through characteristic points ofthe projected display, after which its spatial properties are calculatedgeometrically. To this end, Ashdown's method requires the user toregister the spatial properties of 9 lines, whereas Banning usessubstantially less lines in combination with assumptions on shape,orientation and/or position of the projected display. Both methods,however, rely on either measuring the position of the pointing device oron the requirement that this position remains constant. As such, theyeither rely on sophisticated and expensive tracking equipment orseverely constrain the freedom-of-movement of the user.

Other pointing systems known in the art may be classified as other thanentirely direct-pointing or indirect-pointing systems. Such systemsinclude one disclosed in U.S. Pat. No. 6,417,840 to Daniels (2002),which is combination of a cordless mouse with a laser pointer. Althoughthis system incorporates a direct-pointing device (i.e., the laserpointer), the method used by the system for interacting with thepresentation is indirect (i.e., by means of the mouse) and thereforedoes not afford the fast and more accurate interactive pointing actionsprovided by some other direct-pointing systems described in some of theforegoing publications.

Another system known in the art that uses both direct andindirect-pointing methods is described in U.S. Pat. No. 6,297,804 toKashitani (2001). The disclosed system is a system for pointing to realand virtual objects using the same pointing device. In the disclosedsystem, the pointing means switch between a computer controlled lightsource (e.g., a laser) and a conventional computer cursor, depending onwhether or not the user's intended point-of-aim has crossed a boundarybetween the real and virtual display (i.e., computer-displayed imagery).Various methods are described to establish these boundaries. Althoughthe computer-controlled light source may be regarded as pointing in adirect manner, its point-of-aim is essentially governed by the systemoperator using an indirect-pointing system such as a mouse. Thus, thedisclosed system does not allow for the desired flexibility afforded bytruly direct-pointing methods.

Other systems known in the art include those such as disclosed in U.S.Pat. No. 5,796,386 to Lipscomb et al. (1998), which discloses acalibration procedure to establish the relation between coordinatesystems associated with a handheld device and, in one embodiment, adisplay unit. The system disclosed in the Lipscomb et al. '386 patentmay arguably be applicable as a direct-pointing cursor control system.The disclosed calibration routine requires the user to register at leastthree 3 dimensional positions of the handheld device with a centralprocessing unit. The disclosed system does not appear to includecalibration methods for the case where the display unit is out ofphysical reach of the system operator. The system is thus not practicalfor use at an arbitrary venue.

U.S. Pat. No. 6,084,556 to Zwem (2000) discloses a head-mounted displayunit that displays information governed by the orientation of theapparatus, as measured by a tracking device. This way, the systemcreates the illusion of a large virtual display that is being looked atby the system operator. Of course, the large display alluded to does notconstitute a projected presentation. Also, no methods are disclosed inthe Zwem '556 patent to establish the relative position of thehead-mounted apparatus with respect to the display.

U.S. Patent Application Publication No. 2002/0079143 to Silverstein etal. (2002) discloses a system in which the position of a moveabledisplay with respect to a physical document is established. The '143patent application publication describes calibration routines in whichthe user is required to activate a registration button when the moveabledisplay is over some predetermined position on the physical document.The disclosed system only relates to 2 dimensional applications and,moreover, cannot be used in situations where the interaction region isout of the system operator's physical reach.

SUMMARY OF THE INVENTION

One aspect of the invention is a method for controlling a computerdisplay cursor in an interaction region. A method according to thisaspect of the invention includes projecting an image of the computerdisplay to create the interaction region. A distance is establishedbetween a first point and a second point. The first point has apredetermined relation to the projection device, and the second pointhas a predetermined relation to the interaction region. At least one ofan orientation and a position of a pointing line is measured. Thepointing line has a predetermined relation to a pointing device. Theestablished distance and the at least one of measured position andorientation are used to control the cursor position on the interactionregion.

Another aspect of the invention is a method for controlling a parameterrelated to the position of a cursor on a projected computer screenimage. According to this aspect, the method includes establishing afirst point having a predetermined relation to the projected computerscreen image. An orientation of a pointing line is measured whiledirecting the pointing line to substantially pass through the firstpoint. The pointing line has a predetermined relation to a pointingdevice. A positional coordinate of the pointing device, relative to theprojected computer screen image, is constrained using the measurement ofthe orientation of the pointing line, and the positional coordinate ofthe pointing device is used to control the parameter of said cursor.

Another aspect of the invention is a method for controlling a computerdisplay cursor in an interaction region. The method according to thisaspect includes establishing a first and a second point. Both the firstand the second point have a predetermined relation to the interactionregion. A pointing line with a predetermined relation to a pointingdevice is directed to substantially pass through the first point, andits orientation is measured. The measurement is used to constrain apositional coordinate of the pointing device. The orientation of thepointing line is also measured while directing it to substantially passthrough the second point, and this measurement is also used to constraina positional coordinate of the pointing device. No more than twopositional coordinates of the pointing device are initialized to have apredetermined relation during the first and the second measurement ofthe orientation of the pointing line. The positional coordinates of thepointing device are used. to control the computer display cursor.

Other aspects and advantages of the invention will be apparent from thefollowing description and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a pointing device and base station according to a firstembodiment.

FIG. 2 shows a presentation venue and computer screen.

FIG. 3 shows program steps for selection of appropriate assumptionsaccording to several embodiments.

FIG. 4 shows program steps for construction of information set accordingto several embodiments.

FIG. 5 shows program steps for ascertaining necessary 3D data accordingto several embodiments.

FIG. 6 shows program steps for control of pointing device elements andcomputer cursor according to several embodiments.

FIG. 7 shows one example of a first embodiment.

FIG. 8 shows a second example of one embodiment of the invention.

FIG. 9 shows a third example of one embodiment.

FIG. 10 shows a pointing device and base station according to a secondembodiment.

FIG. 11 shows program steps for construction of information setaccording to several embodiments.

FIG. 12 shows program steps for ascertaining necessary 3D data accordingto several embodiments.

FIG. 13 shows a pointing device and base station according to a thirdembodiment.

FIG. 14 shows program steps for ascertaining necessary 3D data accordingto several embodiments.

FIG. 15 shows projection of interaction structure on horizontal andvertical planes.

FIG. 16 shows an image of calibration point and light spots at thepoints-of-aim.

FIG. 17 shows an alternative embodiment of pointing device.

FIG. 18 shows a prior art: construction of method M for a quadrangle.

FIG. 19 shows a fourth preferred embodiment.

FIG. 20 shows prior art: the correction of vertical and horizontalkeystoning.

FIG. 21 shows application of the cosine rule for non-perpendicularprojection situations.

FIG. 22 shows a pointing device without base station according to afifth embodiment.

FIG. 23 shows a fifth preferred embodiment.

FIG. 24 shows a sixth preferred embodiment.

FIG. 25 shows a seventh preferred embodiment.

DETAILED DESCRIPTION

A first embodiment of the invention will be described with reference toFIG. 1. A pointing device 20 has associated with it a coordinate systemx′ y′ z′. A portable base station 30 has associated with it a coordinatesystem x y z. Pointing device 20 and base station 30 may be equippedwith coordinate sensing devices, 201 and 301, respectively, that enablemeasurement of the 3 dimensional position and 3 dimensional orientationof pointing device 20 and, therefore, of a pointing line 21 (see alsoFIG. 2) that substantially intersects pointing device 20, all measuredrelative to the x y z coordinate system. Pointing line 21 maysubstantially coincide with the long axis of pointing device 20. Forexample, coordinate sensing devices 201 and 301 may be electromagnetictracking devices, such as the 3SPACE FASTRAK® system, manufactured byPolhemus, a Kaiser Aerospace & Electronics Company, Colchester, Vt.Alternatively, coordinate sensing device 201 and 301 may be based on(ultra)sonic tracking systems such as those used in the LOGITECH 2D/6Dcomputer mouse, commercially available from Logitech Inc., Fremont,Calif. Other embodiments for coordinate sensing device 201 and 30 I mayinclude, without limitation, sensors based on LEDs, CCDs,accelerometers; inclinometers, gyroscopes, compasses, magnetometers,etc. Also, combinations of such different types of sensors may be usedto acquire redundant measurements for quality control purposes. Forexample, a controlled-source electromagnetic position tracking systemmay be combined with compasses and inclinometers. Those skilled in theart will appreciate that independent measurements of the Earth'smagnetic field and gravity may be used to enhance the measurements madeby a controlled-source position detecting system. In the invention, anysystem may be used that is capable of determining at least parts of theorientation and position in three dimensions, with respect to coordinatesystem x y z, of a line-segment that substantially intersects pointingdevice 20.

Base station 30 may comprise a plurality of related physical entities,such as described in U.S. Pat. No. 6,608,668 to Faul et al (2003); forclarity of explanation one of these physical entities will be associatedwith coordinate system x y z and be denoted as base station 30, thecenter of which substantially coincides with the origin of coordinatesystem x y z. For purposes of explanation of the invention, the originof coordinate system x′ y′ z′ substantially coincides with the center ofpointing device 20, and the z′-axis is substantially aligned with thelong axis of pointing device 20.

Pointing device 20 may also be provided with a light-beam projector 202,for example a laser. The physical position and orientation of theprojected beam of light with respect to coordinate system x′ y′ z′ maybe established with suitable accuracy at the place of manufacture of thepointing device 20 and may be presumed to be known. For purposes ofexplanation, the beam of light from the light beam projector 202substantially coincides with the z′-axis. Additionally, one or morecontrol buttons 203 a, 203 b or the like may be provided, as well ascommunication and control device 204. Communication and control device204 may be used to control various features of pointing device 20 andmay also communicate via wire, or wirelessly, with base station 30, acentral processing unit (not shown separately) such as a COMPAQ ArmadaM700 as manufactured by Hewlett Packard Company, Palo Alto, Calif., orany other components of the system. The central processing unit may alsocontrol a presentation with which user interaction is desired. Forclarity of the description which follows, the central processing unitwill be referred to hereinafter as “the computer.” Furthermore, pointingdevice 20 may include an internal power source 205, for example abattery, or instead may be connected such as by wire to an externalpower source.

In addition to coordinate sensing device 301, base station 30 may beprovided with communication and control device 304. Communication andcontrol device 304 may be used to control various features of basestation 30. Communication and control device 304 may communicate withpointing device 20, with the computer (not shown) that may control thepresentation images, or with any other components of the system. Thecommunication and control device 304 may use wire- or wirelesstechnology. In the present embodiment communication with the computeroccurs via a universal serial bus (USB) compatible device or the like(not shown). Communication and control device 304 may communicatewirelessly with a relay device (not shown) that communicates via a USBcompatible device or the like with the computer (not shown) that maycontrol the presentation with which user-interaction is desired. Theremay be visible markings 302 on base station 30 that indicate theorientation of one or more coordinate planes defined by prescribedrelations between coordinates x, y, z, as well as features of theposition of the origin of the x y z coordinate system. For example, aline may be used to indicate the z-x-plane of the x y z coordinatesystem, for which the y-coordinate is zero; another line may be used toindicate the z-y-plane, for which the x-coordinate is zero. Base station30 may be provided with a level-sensing device 303 to determine whetheror not one of the coordinate planes of coordinate system x y z issubstantially horizontal. Level-sensing device 303 may measure twoangles or equivalent characteristics that define the orientation of oneof the coordinate planes of coordinate system x y z with respect to ahorizontal surface. Level-sensing device 303 can be a device such asdisclosed in U.S. Pat. No. 6,466,198 to Feinstein (2002), which makesuse of model ADXL202 accelerometers sold by Analog Devices Inc.,Norwood, Mass. The disclosed accelerometers provide tilt angleinformation depending on their inclination relative to Earth's gravity.It will be apparent to those skilled in the art that many other types ofsensors may be used as embodiment for level-sensing device 303, forexample, capacitance-type bubble levels. See, for example, U.S. Pat. No.5,606,124 to Doyle et al. Finally, base station 30 may be equipped witha power source 305, for example a battery, or may have a connection toan external power source.

Referring to FIG. 2, a projection device 40 is arranged to project animage 50 generated by, for example, the computer (not shown). Theprojection device 40 may be used to generate a projection image 70 on aprojection region 60. For example, projection device 40 may be a 2000lumen projector, model XL8U, manufactured by Mitsubishi Corp. Projectionregion 60 may be a surface, such as a wall or a projection screen. Theprojection region 60 may define a flat plane or a may define a moreelaborate 2 dimensional (2D) or even 3 dimensional (3D) structure. InFIG. 2, projection region 60 is shown as a screen, but this is only forpurpose of illustrating the principle of the invention and is notintended to limit the scope of the invention. Alternatively thecombination of projection device 40 and projection region 60 may beincorporated in one and the same physical device, such as a televisionreceiver (cathode ray tube display), liquid crystal display (LCD) screenor the like.

There is a region of space that is designated as a region on whichinteraction with the user is desired. This region is denoted as theinteraction region 71. The interaction region 71 may be flat (planar) ormay be a more elaborate 2D or even 3D structure. The interaction region71 may have features in common with projection image 70 and may beassociated in some way with a computer screen interaction region 51. Inthe present embodiment, however, it will be assumed that interactionregion 71 or a scaled version thereof, is substantially part of orsubstantially coincides with projection image 70. Moreover, it will beassumed in this embodiment that interaction region 71 substantiallycoincides with the projection of computer screen interaction region 51as projected by projection device 40.

For display systems which include a separate projection device 40 andprojection region 60, the optical axis of projection device 40 may notbe aligned with any vector normal to projection region 60. Moreover,projection region 60 may not be flat and 14 therefore may not even be a2D shape. Consequently, projection image 70 and interaction region 71may in general not be flat or rectangular, even if the imagery generatedby the computer is scaled so as to be presented in rectangular form. Inthis embodiment, however, it is assumed that projection region 60,projection image 70 and interaction region 71 are substantially flat.Furthermore, it is assumed in this embodiment that interaction region 71is substantially a quadrangle and that the associated computer screeninteraction region 51 is substantially rectangular.

Additionally, calibration points 721 a, 721 b, 721 c, 721 d may beprovided that may define characteristic features of interaction region71. For example, interaction region 71 may be trapezoidal in shape, inwhich case calibration points 721 a, 721 b, 721 c, 721 d may definecorners of interaction region 71 or corners of a scaled version thereof.Furthermore, screen marks 521 a, 521 b, 521 c, 521 d may be provided,and may but need not be associated with calibration points 721 a-721 dFor example, calibration points 721 a-721 d may coincide with theprojected versions of screen marks 521 a-521 d and may in this way beidentified by projection device 40. Calibration points 721 a-721 d mayalso be identified by other means than projection, for example by uniquedescriptions such as the ‘upper right corner of interaction region 71’,‘center of interaction region 71’ etc.

The operation of the present embodiment will now be described withreference to FIGS. 1, 2 and 3. A display system is set up at the venuewhere a presentation is to made, which can be a combination of aportable projection device 40 and projection surface 60, for example awall. The display system is connected, using appropriate interconnectiondevices, to the computer (which may be in the base station 30 or locatedelsewhere) that generates the presentation images.

The system user positions base station 30 at a convenient location,preferably not far from where the user intends to conduct thepresentation. The user may position base station 30 in such a way thatone of the coordinate planes of the x y z coordinate system issubstantially parallel or substantially coincident with projectionregion 60. The visual markings 302 may assist in such positioning.Subsequently, the user connects base station 30 to the computer (notshown), for example via a USB compatible device connection (not shown),or using a wireless relay device (not shown). The computer may bedisposed in the base station 30 in some embodiments. In someembodiments, the computer may recognize the base station connection andstart a program, part of which may be contained in the communication andcontrol device 304, in communication and control device 204, or incontrol logic contained in the wireless relay device (not shown).Alternatively, the user may be required to load the program into thecomputer manually via a CD-ROM drive, a floppy drive, memory stick (USBmass storage device not shown) or the like. However it is loaded intothe computer, the program may initiate a calibration routine that has asits object establishment of the shape, position, size and orientation ofa defined interaction structure 72 relative to the x y z coordinatesystem. The interaction structure 72 is assumed to substantiallycoincide with interaction region 71. The operation of the program willnow be explained with reference to FIG. 3.

At 80 a the program is initiated. During step 80 b a default assumptionis made regarding interaction region 71. Specifically, interactionregion 71 is assumed to substantially coincide with a well-definedinteraction structure 72 (FIG. 2 shows an interaction region 71 and aninteraction structure 72 that are clearly not identical; this differenceis however only meant as an example, and as a practical matter ispreferably as small as possible). At 80 b default values are entered forthe orientation and position of this interaction structure 72. Forexample, the default setting may assume interaction region 71 tosubstantially coincide with an interaction structure 72 that is a (flat)quadrangle positioned in a vertical plane that substantially intersectsthe origin of the x y z coordinate system associated with base station30. As another example, the default setting may provide that interactionregion 71 substantially coincides with an interaction structure 72 thatis an isosceles trapezoid of which the parallel edges are horizontal andof which the position is unknown. Using such default values, calibrationpoints 721 a-721 d not only define characteristic features ofinteraction region 71 but also of interaction structure 72.

At 80 c a decision is made whether the default values for interactionregion 71 and interaction structure 72 should be accepted or overriddenby the user. In making this decision, input from level-sensing device303 and visible markings 302 on base station 30 may be used. If thedefaults are to be accepted, the program continues to 80 j, the detailsof which are explained below with reference to FIG. 4. If the defaultsare overridden, the program continues at 80 d to 80 i, during each ofwhich the user may override any of the default settings. The user may beaided during any of 80 d to 80 i by a library of predetermined shapes,predetermined orientations and/or predetermined positions, or the usercan be provided by the program with the capability to construct customshapes, orientations and/or positions. In any case, the programcontinues to 80 j.

It will be appreciated by those skilled in the art that once anassumption has been made regarding the shape of interaction structure72, it is possible to construct a set of three dimensionally distributedpoints in space that completely determines the 3D position, orientationand size of the interaction structure 72. The number of points in theset will depend on the complexity of the assumed shape and the ingenuitywith which the points are chosen. For example, a rectangular shape thatis arbitrarily oriented in space is completely determined by a set of 3points that coincide with 3 of its corners, but is also completelydetermined by a set of 8 points, pairs of which may determine each ofthe four edges of the rectangle.

Referring to FIG. 4, describing details of program element 80 j, thecomputer program continues at 90 a, at which a set P is generated thatincludes a quantity n of points in space, each represented as C(i)(0<i<n+1), that uniquely determine interaction structure 72, togetherwith descriptions of their roles in determining this structure. Forexample, if the program elements outlined in FIG. 3 reveal thatinteraction region 71 is assumed rectangular, set P may hold threepoints described as the upper-left corner, the lower-left corner and theupper-right corner of a rectangular interaction structure 72. If,alternatively, the projection of computer screen interaction region 51is substantially rectangular, and this projected rectangle has the samecenter but is physically two times smaller than interaction structure72, the three points may be described as the upper-right corner, thelower-left corner and the lower-right corner of a rectangle that has thesame center but is two times smaller than interaction structure 72.Thus, by carefully choosing the points in set P together with theirdescription, any interaction structure 72 may be completely determined.

In addition to set P, a 3D point CA and another 3D point CB aredetermined at 90 a. These additional 3D points are determined so as tolie away from interaction region 71, that is, they lie at some distanceout of the plane that is closest to or substantially containsinteraction region 71. The distance may be comparable to an edge ofprojection image 70. For example, point CA may be determined to lie nearprojection device 40 and point CB may be determined to lie at a distancefrom point CA that may be equal to the distance between projectiondevice 40 and projection surface 60, measured substantially parallel toprojection surface 60. Other choices for point CA and point CB may alsobe used. Additionally, sets A and B are generated during step 90 a. SetA includes a number n of lines A(i), each of which connects point CA toone of the points C(i) (0<i<n+1) in set P. Likewise, set B holds anumber n of lines B(i), each of which connects point CB to one of thepoints C(i) (0<i<n+1) in set P. Finally, at 90 a, counters a, b are bothinitialized to the value of n.

Flow then continues on to step 90 b, where a decision is made whetherthe program elements outlined in FIG. 3 have revealed any information onthe 3D position and orientation of interaction structure 72. If thedecision is positive flow continues to step 90 c, where this informationis used to establish a priori relationships between the coordinates ofthe points in set P. For example, the steps outlined in FIG. 3 may haverevealed that interaction structure 72 is assumed to coincide with thex-z plane, in which case the a priori relationships may include therequirement that the y-coordinates of all points in set P are equal tozero. A pre-set library of a priori relationships may be provided to aidin the execution of this program element.

The program continues to 90 d, which may also be reached from 90 b ifthe decision at 90 b is negative. At 90 d, line B(b) is removed from setB, after which counter b is decremented by 1.

The program continues to 90 e, where a decision is made whether complete3D information on the lines in sets A and B, together with a prioriinformation, constitutes enough information to uniquely determine thecoordinates of all the points in set P. For example, the programelements outlined in FIG. 3 may have determined that interaction region71 is assumed to be rectangular, but no information is known regardingits orientation or position. In this case set P may contain threepoints, corresponding to three corners of a rectangular interactionstructure 72. Each point C(i) (0<i<4) in set P would then be uniquelydetermined by the intersection of a line from set A and a line from setB if and only if sets A and B contained three lines each.

If the decision at 90 e is negative, the program continues to 90 f,where line B(b+1) is added once more to set B.

If the decision at 90 e is positive, program flow continues to 90 g. Theprogram flow may also continue from 90 f to 90 g. At 90 g a decision ismade whether counter b has reached zero, in which case the lines left inset B (which number may be equal to zero) are deemed necessary forunique determination of the coordinates of all the points in set P. Ifthe decision at 90 g is positive, program flow continues to 90 h. If thedecision at 90 g is negative, program flow reverts to 90 d.

During 90 h, line A(a) is removed from set A, after which counter a isdecremented by 1. Program flow then continues to 90 i, where a decisionis made whether complete 3D information on the lines in sets A and B,together with the a priori information, constitute enough information touniquely determine the coordinates of all the points in set P.

If the decision at 90 i is negative, program flow continues to step 90j, where line A(a+1) is added once more to set A.

If the decision at 90 i is positive, program flow continues to step 90k. The program flow also continues from 90 j to 90 k. At 90 k, adecision is made whether counter a has reached zero, in which case thelines left in set A (which number may be equal to zero) are deemednecessary for unique determination of the coordinates of all the pointsin set P. If the decision at 90 k is negative, program flow reverts to90 h. If the decision at 90 k is positive, program flow continues to 90m, the details of which are described with reference to FIG. 5.

In FIG. 5, the program continues to 100 a, where counter p isinitialized to 1 and variable n is set to the number of points in set P.

Program flow then continues to 100 b where a decision is made whetherline A(p) (connecting point CA to point C(p)) is included in set A. Ifthe decision is negative, program flow continues to 100 c, where counterp is incremented by 1.

If the decision at 100 b is positive, program flow continues to 100 e,at which the program identifies point C(p) to the user and then canrequire the user to point to C(p) using pointing device 20,advantageously (but not necessarily) aided by light-beam projectiondevice 202, preferably from a position such as point CA, as determinedpreviously at 90 a (FIG. 4). The user can also be queried to affirm thepointing action by, for example, activating one of the buttons 203 a or204 b, or the like. The identification of point C(p) may occur, forexample, by having the program display a visible screen mark 521 a-521 dat a position on computer screen interaction region 51 associated withC(p), which may then be projected by projection device 40 to coincidewith point C(p). Other means of identification are also possible, suchas the characterization of point C(p) as ‘upper-right corner’ or thelike.

Program flow then continues on to 100 f, where the program can waituntil the highlighting action is affirmed by the user, after which the3D orientation of the z′-axis and the 3D position of the z′=0 point (ofthe z′-axis) are measured with respect to the x y z coordinate system,using coordinate sensing device 201 and 301, and communicated to thecomputer using communication and control device 204 and/or 304. This 3Dorientation and position is then associated with line A(p) and stored inmemory. Program flow then continues to 100 c.

After 100 c program flow continues to 100 d where a decision is madewhether p is equal to n+1. If the decision is positive, it can beconcluded that all necessary data on lines A(p) have been ascertainedand program flow can continue to 100 g. If the decision at 100 d isnegative, program flow reverts to 100 b.

At 100 g a decision is made whether set B is empty. If the decision ispositive, it can be concluded that enough information to uniquelydetermine the 3D coordinates of all the points in set P is contained inthe a priori relationships combined with the available data on the linesin set A, and program flow continues to 100 p. If the decision at 100 gis negative, program flow continues to 100 h.

At 100 h counter p is again initialized to 1. The user is subsequentlyrequired to reposition the pointing device 20 to a different location,displacing it substantially parallel to projection region 60 over adistance substantially equal to the distance between projection region60 and his or her present location. Other locations may also be used.

Flow then continues to 100 i where a decision is made whether line B(p)(connecting point CB to point C(p)) is included in set B. If thedecision is negative, program flow continues to 100 j where counter p isincremented by 1.

If the decision at 100 i is positive, program flow continues to 100 m,at which point the program identifies point C(p) to the user and canquery the user to highlight this point using light-beam projectiondevice 202. The program can also query the user to affirm thehighlighting by, for example, activating one of the buttons 203 a or 204b or the like. The identification of point C(p) may occur, for example,by having the program display a visible screen mark 521 a, 521 b, . . .at a position on computer screen interaction region 51 associated withC(p), which may then be projected by projection device 40 to coincidewith point C(p). Other means of identification are also possible, suchas the characterization of point C(p) as ‘upper-right corner’ or thelike.

Program flow then continues to 100 n, where the program may wait untilthe highlighting action is affirmed. After affirmation, the 3Dorientation of the z′-axis and the 3D position of the point z′=0 (of thez′-axis) are measured with respect to the x y z coordinate system, usingcoordinate sensing device 201 and 301, and are communicated to theprogram using communication and control device 204 and/or 304. The 3Dorientation and position are then associated with line B(p) and can bestored in memory. Program flow then continues to 100 j.

After 100 j program flow continues to 100 k where a decision is madewhether p is equal to n+1. If the decision is positive, it is concludedthat all necessary data for lines B(p) has been ascertained and programflow continues to 100 p. If the decision is negative, program flowreverts to 100 i.

At 100 p it is concluded that the combined information of the a priorirelationships and the data for lines in sets A and B that is stored inmemory is enough to establish the coordinates of all points in P, as isfurther explained with reference to FIG. 6.

Referring to FIG. 6, as will be appreciated by those skilled in the art,at 110 a the available information contained in the a priorirelationships and the orientations and positions of the lines in sets Aand B may be used to establish the coordinates of each of the points inset P, all measured with respect to the x y z coordinate system. It mayhappen that the 3D data associated with lines A(p) and B(p) (I<p<n+l)cause them not to intersect at point C(p); for example, this could bethe case due to insufficient accuracy in determining the orientationsand positions of lines A(p) and B(p). Under such circumstances pointC(p) may be estimated or determined in such a way that it lies halfwayalong the shortest line-segment that connects lines A(p) and B(p). Othermethods for determining point C(p) are also possible.

Once the complete 3D description of interaction structure 72 has beenestablished (given the available data and assumptions), program flowcontinues to 110 b. At 110 b, a method M is constructed that maps pointsof interaction structure 72 to computer screen interaction region 51.Such methods are well known to those skilled in the art. For example, amethod such as that described in U.S. Pat. No. 6,373,961 to Richardsonet al. (2002) may be used, but other appropriate methods may also beused. For completeness, the method disclosed in the Richardson et al.'961 patent will briefly be explained here. Referring to FIG. 18,interaction structure 72 may be defined by line segments JK and ML.Furthermore, point F may define point-of-aim 210, which is also theintersection between the z′-axis and interaction structure 72. The linesegments JK and ML are generally not parallel. If not parallel, then theline segments JK, ML are extended until they meet at a point H. Then aline is drawn through F and H, that intersects segment JM at point P andsegment KL at point 0. If the segments JK and ML are parallel, the aline is drawn instead through F that is parallel to JK, and whichintersects the other two line segments as described. The processcontinues similarly with the other segments, which are extended to meetat point G. A line through F and G intersects segment JK at point R andsegment ML at point Q. Subsequently, the following distances aremeasured and noted as percentages: JRJJK, KO/KL, LQ/LM and MP/MJ. Thenfour points S, T, U, V are chosen as shown in FIG. 18, that form anexact rectangle STUV; this rectangle may define computer screeninteraction region 51. Further, points W, X, Y, Z are chosen on thesides of the rectangle STUV in such a way that SZ/ST=JRJJK,TW/TU=IKO/KL, UX/UV=LQ/LM and VYNS=MP/MJ. Then the points Z and X arejoined by a line, and the points Y and W are joined by a line. The pointof the intersection of these latter two lines is N, which is the pointthat results from the operation of method M.

Referring once again to FIG. 6, after 110 b, program flow continues to110 c where a decision is made whether the user has requested theprogram to end. If the decision is positive, the program ends at 110 k.

If the decision at 110 c is negative, program flow continues to 110 d,where a decision is made regarding whether the user has requested adirect-pointing action, such as activating button 203 a or 203 b or thelike.

If the decision at 110 d is negative, program flow continues to 110 e,where the light-beam projection device 202 can be instructed or causedto de-activate (using, for example, control device 204 and/or 304).Furthermore, a software routine (not shown) that controls computercursor 501 (see also FIG. 2) is instructed that the user does not wantto execute a direct-pointing action, after which program flow reverts to110 c. Such cursor control routines are well known to those skilled inthe art and are therefore not described here.

If the decision at 110 d is positive, program flow continues to 110 f,where a decision is made whether the z′-axis intersects interactionstructure 72. Those skilled in the art will appreciate that this ispossible because all relevant 3D information is known or is measurableby coordinate sensing device 201 and 301. If the decision is positive,program flow continues to 110 h, at which method M is used to map theintersection point of the z′-axis with interaction structure 72 tocomputer screen interaction region 51. This mapped position, as well asthe user's desire to execute a direct-pointing action are communicatedto the cursor control routine (not shown) running on the computer.

After 110 h, program flow continues to 110 i where a decision is madewhether the user wishes to execute an action associated with theposition of cursor 501. For example, activating button 203 a or 203 b orthe like may indicate such a request. If the decision is negative,program flow reverts to 110 c.

If the decision at 110 i is positive, program flow continues to 110 j,where the cursor control routine (not shown) is requested to execute theintended action. Such an action may, for instance, be a ‘double-click’action or the like. Program flow then reverts to 110 c.

There may be situations for which the system of equations that allowsdetermination of all coordinates of the points in set P will be somewhatill-posed. This could for example occur if locations CA and CB arechosen too close together, or if the angles of some pointing lines 21with respect to interaction region 71 are too small, as will beappreciated by those skilled in the art. In such cases, the user may bedirected to choose a different point for locations CA and/or CB fromwhere the various points C(p) are to be highlighted.

Thus, methods and means are disclosed that afford a highly flexible andeasily deployable system for interacting with a presentation in adirect-pointing manner at a location not specifically designed oradapted for such a purpose. Moreover, although it is desirable to havehighly accurate coordinate sensing device 201 and 301, in some instancessuch may not be necessary because the visual feedback afforded by adisplayed cursor will compensate to a large extent for any errors madein the measurements of coordinate parameters. The same holds true forthe importance of any discrepancies between the actual shape ofinteraction region 71 and that of the interaction structure 72 that isassumed to coincide with it. Also, minor discrepancies

between the actual position and orientation of interaction region 71 andthe assumed position and orientation of interaction structure 72 (whichis assumed to coincide with interaction region 71) need not be ofcritical importance, as will be appreciated by those skilled in the art.

Mathematically, there are infinite possible shapes for interactionstructure 72 as well as infinite a priori relationships, sets P, A, Band methods M. To further explain the invention, what follows is a listof examples that are believed will be encountered most often. Theseexamples are not meant to restrict the scope of the present invention inany way, but merely serve as further clarification.

Referring to FIG. 7, it can be assumed that the venue where thepresentation is to be made is set up such that projection image 70 andinteraction region 71 are substantially rectangular. Such may be thecase when the optical axis of projection device 40 is substantiallyparallel to the normal to projection region 60. Projection region 60 maybe a projection screen on a stand, as shown, but may also be a wall orany other substantially vertical surface. It can also be assumed thatthe bottom and top edges of both projection image 70 and interactionregion 71 are substantially horizontal. Furthermore, it can be assumedthat interaction region 71 is substantially the same size as projectionimage 70 and that computer screen interaction region 51 is substantiallythe same size as computer screen image 50. Additionally, it can beassumed that calibration points 721 a, 721 d substantially coincide withthe projected versions of screen marks 521 a, 521 d. Finally, it isassumed that base station 30 is positioned such that the x-y-plane issubstantially horizontal and the x-z-plane substantially coincides withthe plane of interaction region 71. To facilitate in the positioning ofbase station 30, the user may be aided by level-sensing device 303 andvisible markings 302.

Referring to FIGS. 3 and 7, process elements 80 a-j may then result,either through default settings or through user-supplied settings, inthe assumptions that interaction structure 72 is a rectangle that liesin the x-z plane and that its top and bottom edges are parallel to thex-y plane.

Referring to FIGS. 4 and 7, program element 90 a may then result in aset P containing three points that define the upper-right corner C(l),the upper-left corner C(2) and the lower-left corner C(3) of interactionstructure 72. Additionally, program element 90 a may determine point CA(not denoted. as such in FIG. 7) to lie away from projection region 60,for example at the center of one of the two instantiations of pointingdevice 20, as shown in FIG. 7. Point CB may be determined anywhere inspace. Program element 90 a may also result in sets A and B eachcontaining three lines, connecting the points in set P to points CA andCB respectively. Program element 90 c may then result in a priorirelationships such that all points in set P have y-coordinate equal tozero, that the upper-left corner will have an x-coordinate equal to thex-coordinate of the lower-left corner and that its z-coordinate will beequal to the z-coordinate of the upper-right corner. It will then beappreciated by those skilled in the art that this a priori information,together with complete 3D information on two lines in set A will besufficient to uniquely determine the position, size and orientation ofinteraction structure 72 with respect to the x y z coordinate system.Therefore, program elements 90 d, 90 e, 90 f, 90 g may result in anempty set B. Moreover, program elements 90 h, 90 i, 90 j, 90 k mayresult in the removal from set A of line A(2), connecting point CA topoint C(2).

Referring to FIGS. 5 and 7, program elements 100 a-p may result in theprogram identifying point C(1) by instructing the computer (not shown)and projection device 40 to project screen mark 521 a onto projectionregion 60, resulting in the appearance of calibration point 721 a.Subsequently, the user is required to use light-beam projection device202 to highlight calibration point 721 a and indicate the success ofthis action to the computer (not shown) by, for example, pressing button203 a. As a result, the orientation and position of the z′-axis areassumed to define line A(1). The same actions are then performed forpoint C(3) and line A(3). Since set B is empty, the user need not berequired to reposition pointing device 20. The two depictions ofpointing device 20 in FIG. 7 are meant to illustrate that pointingdevice 20 only needs to be redirected and not repositioned during thisexercise. It will be appreciated by those skilled in the art thatprogram element 100 p will then successfully result in a full 3Ddescription of the three points in set P.

Referring to FIGS. 6 and 7, program element 110 a may then result in thedescription of interaction structure 72 as a rectangle with upper-rightcorner C(l), upper-left corner C(2) and lower-left corner C(3). Step 110b may then result in a method M that maps a point 6 (not shown) ininteraction structure 72 to a point E (not shown) in computer screeninteraction region 51 in such a way that the ratio of distances between6 and any two of the four corners of interaction structure 72 is thesame as the ratio of distances between s and the two correspondingcorners of computer screen interaction region 51, as will be appreciatedby those skilled in the art. Steps 110 c-k may then result in a veryintuitive cursor control device that responds to a direct-pointingaction by de-activating light-beam projection device 202 and showingcursor 501 at substantially point-of-aim 210 of pointing device 20whenever point-of-aim 210 lies in projection image 70 (which in thisexample, by assumption, substantially coincides with interaction region71). Since there is no necessity for a computer generated cursor and alight spot to be visible simultaneously there will not be any cause forconfusion as to which of the two is the ‘actual cursor’. Cursor 501 maybe hidden from view when point-of-aim 210 does not lie in interactionstructure 72, in which case light-beam projection device 202 may beactivated automatically. Also, intuitive actions such as ‘double click’may be provided by steps 110 c-k.

Referring to FIG. 8, it can be assumed that the venue where thepresentation is to be given is set up such that projection image 70 andinteraction region 71 are substantially rectangular. Such may be thecase when the optical axis of projection device 40 is substantiallyparallel to the normal to projection region 60. Projection region 60 maybe a projection screen on a stand, as shown, but may also be a wall orany similar surface. Projection region 60 can be assumed to be orientedsubstantially vertically. It can also be assumed that the bottom and topedges of both projection image 70 and interaction region 71 aresubstantially horizontal. Furthermore, it can also be assumed thatinteraction region 71 is substantially the same size as projection image70 and that computer screen interaction region 51 is substantially thesame size as computer screen image 50. Additionally, it can be assumedthat calibration points 721 a, 721 c, 721 d substantially coincide withthe projected versions of screen marks 521 a, 521 c, 521 d. Finally, itcan be assumed that base station 30 is positioned such that the xy-planeis substantially horizontal. No assumptions are made regarding thex-z-plane and the y-z-plane other than that they are substantiallyvertical. To facilitate in the positioning of base station 30, the usermay be aided by level-sensing device 303.

Referring to FIGS. 3 and 8, program elements 80 a-j may then result,either through default settings or user-supplied settings, in theassumptions that interaction structure 72 is a rectangle of which thetop and bottom edges are parallel to the x-y-plane (i.e., they arehorizontal) and the left and right edges are perpendicular to thex-y-plane (i.e., they are vertical).

Referring to FIGS. 4 and 8, program element 90 a may then result in aset P containing three points that define the upper-left corner C(1),the upper-right corner C(2) and the lower-left corner C(3) ofinteraction structure 72. Additionally, program element 90 a maydetermine point CA (not denoted as such in FIG. 8) to lie away fromprojection region 60, for example, at the center of one of the threeinstantiations of pointing device 20, as shown in FIG. 8. Point CB (notdenoted as such in FIG. 8) may be determined to be displaced from pointCA in a direction substantially parallel to interaction region 71, overa distance that may be the same order of magnitude as the size ofinteraction region 71. Program element 90 a may also result in sets Aand B each containing three lines, connecting the points in set P topoints CA and CB respectively. Program element 90 c may then result in apriori relationships requiring that the upper-left corner will have x-and y-coordinates equal to the x- and y-coordinates of the lower-leftcorner and that its z-coordinate will be equal to the z-coordinate ofthe upper-right corner. As will be explained in detail below, the apriori information together with complete 3D information on two lines inset A and one line in set B will be enough to uniquely determine theposition, size and orientation of interaction structure 72 with respectto the x y z coordinate system. Therefore, program elements 90 d, 90 e,90 f, 90 g may result in set B containing only line B(3). Moreover,program elements 90 h, 90 i, 90 j, 90 k may result in set A containingonly lines A(1) and A(2).

Referring to FIGS. 5 and 8, program elements 100 a-p may result in theprogram identifying point C(1), C(2) and C(3) by means of projectiondevice 40, in a manner similar to the one described in the previousexample. In this case, however, C(1) and C(2) may be highlighted fromapproximately the same location CA, but C(3) needs to be highlightedfrom a different” location CB. To explain that the above-mentionedinformation is enough to uniquely establish the position, size andorientation of interaction structure 72, define the points in set P as:

$\begin{matrix}{{C(1)} = \begin{pmatrix}{x_{1} + {\lambda_{1} \cdot {Rx}_{1}}} \\{y_{1} + {\lambda_{1} \cdot {Ry}_{1}}} \\{z_{1} + {\lambda_{1} \cdot {Rz}_{1}}}\end{pmatrix}} & (1) \\{{C(2)} = \begin{pmatrix}{x_{1} + {\lambda_{2} \cdot {Rx}_{2}}} \\{y_{1} + {\lambda_{2} \cdot {Ry}_{2}}} \\{z_{1} + {\lambda_{2} \cdot {Rz}_{2}}}\end{pmatrix}} & (2) \\{{C(3)} = \begin{pmatrix}{x_{3} + {\lambda_{3} \cdot {Rx}_{3}}} \\{y_{3} + {\lambda_{3} \cdot {Ry}_{3}}} \\{z_{3} + {\lambda_{3} \cdot {Rz}_{3}}}\end{pmatrix}} & (3)\end{matrix}$

Here, points CA and CB are defined as (x₁, y₁, z₁) and (x₃, y₃, z₃)respectively. Moreover, lines A(1), A(2) and B(3) are defined as passingthrough CA, CA and CB respectively, lying in directions governed by(Rx₁, Ry₁, Rz₁), (Rx₂, Ry₂, Rz₂) and (Rx₃, Ry₃, Rz₃). All of thesequantities are presumed to be measured by coordinate sensing device 201and 301. For a unique description of the points in set P a solution isrequired for λ₁, λ₂ and λ₃.

Using these definitions the conditions described above can be writtenas:

$\begin{matrix}{{\begin{bmatrix}{Rz}_{1} & {- {Rz}_{2}} & 0 \\{Rx}_{1} & 0 & {- {Rx}_{3}} \\{Ry}_{1} & 0 & {- {Ry}_{3}}\end{bmatrix}\begin{pmatrix}\lambda_{1} \\\lambda_{2} \\\lambda_{3}\end{pmatrix}} = \begin{pmatrix}0 \\{x_{3} - x_{1}} \\{y_{3} - y_{1\;}}\end{pmatrix}} & (4)\end{matrix}$

which can be solved in a straightforward manner according to theexpression:

$\begin{matrix}{\begin{pmatrix}\lambda_{1} \\\lambda_{2} \\\lambda_{3}\end{pmatrix} = \begin{pmatrix}\frac{\lbrack {{{Rx}_{3} \cdot ( {y_{3} - y_{1}} )} + {{Ry}_{3} \cdot ( {x_{1} - x_{3}} )}} \rbrack}{\lbrack {{{Rx}_{3} \cdot {Ry}_{1}} - {{Rx}_{1} \cdot {Ry}_{3}}} \rbrack} \\\frac{\lbrack {{Rz}_{1}/{Rz}_{2}} \rbrack \cdot \lbrack {{{Rx}_{3} \cdot ( {y_{3} - y_{1}} )} + {{Ry}_{3} \cdot ( {x_{1} - x_{3}} )}} \rbrack}{\lbrack {{{Rx}_{3} \cdot {Ry}_{1}} - {{Rx}_{1} \cdot {Ry}_{3}}} \rbrack} \\\frac{\lbrack {{{Rx}_{1} \cdot ( {y_{3} - y_{1}} )} + {{Ry}_{1} \cdot ( {x_{1} - x_{3}} )}} \rbrack}{\lbrack {{{Rx}_{3} \cdot {Ry}_{1}} - {{Rx}_{1} \cdot {Ry}_{3}}} \rbrack}\end{pmatrix}} & (5)\end{matrix}$

Note that this solution shows that, if point C(3) was highlighted fromany point on the pointing line 21 that is used to highlight point C(1),i.e.,

$\begin{matrix}{\begin{pmatrix}x_{3} \\y_{3} \\z_{3}\end{pmatrix} = \begin{pmatrix}{x_{1} + {\tau \cdot {Rx}_{1}}} \\{y_{1} + {\tau \cdot {Ry}_{1}}} \\{z_{1} + {\tau \cdot {Rz}_{1}}}\end{pmatrix}} & (6)\end{matrix}$

the solution for the three unknown A's would collapse to

$\begin{matrix}{\begin{pmatrix}\lambda_{1} \\\lambda_{2} \\\lambda_{3}\end{pmatrix} = \begin{pmatrix}\tau \\{\tau \; {{Rz}_{1}/{Rz}_{2}}} \\0\end{pmatrix}} & (7)\end{matrix}$

making unique determination of interaction structure 72 impossible.Conversely, the fact that points C(1) and C(2) are, in this example,both highlighted from substantially the same point CA does not cause anyproblems. It will be appreciated by those skilled in the art that C(1)and C(2) may also be highlighted from different points, without loss offunctionality.

Referring to FIGS. 6 and 8, the course and results of program elements110 a-k will be similar to those described in Example 1.

Referring to FIG. 9, in another example, the only assumptions made arethat interaction region 71 is substantially the same size as projectionimage 70, computer screen interaction region 51 is substantially thesame size as computer screen image 50, calibration points 721 a, 721 b,721 c, 721 d substantially coincide with the. Projected versions ofscreen marks 521 a, 521 b, 521 c and 521 d and computer screeninteraction region 51 is substantially rectangular. This situation mayarise when the optical axis of projection device 40 is not aligned withthe normal to projection surface 60.

Referring to FIGS. 3 and 9, program elements 80 a-j may then result,either through default settings or user supplied settings, in theassumption that interaction structure 72 is a quadrangle; no assumptionsare made regarding its position or orientation.

Referring to FIGS. 4 and 9, program elements 90 a may then result in aset P containing four points that define the upper-right, upper-left,lower-right and lower-left corners of interaction structure 72.Additionally, program element 90 a may conceive point CA (not denoted assuch in FIG. 9) to lie away from projection region 60, for example atthe center of the first of the two instantiations of pointing device 20,as shown in FIG. 9. Point CB may, for example, be determined to lie atthe center of the second of the two instantiations of pointing device20, as drawn in FIG. 9. Now, program elements 90 d, 90 e, 90 f, 90 g andsteps 90 h, 90 i, 90 j, 90 k may result in sets A and B each containing4 lines.

Referring to FIGS. 5 and 9, program elements 100 a-p may result in theprogram identifying the four points in set P by means of projectiondevice 40, in a manner similar to the one described in Example 1. Inthis case, however, all four points in set P may be highlighted fromapproximately the same location CA first, after which all four pointsmay be highlighted from approximately the same location CB. Note thatFIG. 9, for reasons of clarity, only depicts the highlighting ofcalibration point 721 a. Each point C(i) may then be constructed as thepoint lying halfway the shortest line segment connecting lines A(i) andB(i); this line segment may have zero length.

Referring to FIGS. 6 and 9, the course and results of program elements110 a-k will be similar to those described with respect to the firstexample, with the exception of program element 110 b, the mappingelement. Now, a more elaborate method M is needed than the one describedin previous examples. For example, a method such as described in U.S.Pat. No. 6,373,961 to Richardson (2002) may be utilized, but otherappropriate methods may also be used.

A second embodiment will now be made with reference to FIG. 10, whichshows pointing device 20 and base station 30. Here, in addition to anyor all of the elements mentioned in the previous embodiment, pointingdevice 20 is also provided with distance measuring device 206, theposition and orientation of which with respect to the x′ y′ z′coordinate system may be ascertained at the place of manufacture thereofand will be presumed to be known. Distance measuring device 206 may forexample be embodied by a focus adjustment, or optical sensor. A digitalcamera may also be used. Distance measuring device 206 may determine thepoint-of-aim-distance 211 between the origin of the x′ y′ z′ coordinatesystem and the point-of-aim, measured substantially parallel to thez′-axis (see also FIG. 2). The point of aim 210 may lie on projectionregion 60, on which interaction region 71 may lie. Distance measuringdevice 206 may have associated with it a manual focus adjustment thatmay be adjusted by the user (manual focus adjustment not shown).Alternatively, distance measuring device 206 may comprise a circuit (notshown) that automatically determines the point-of-aim-distance 211between the origin of the x′ y′ z′ coordinate system and thepoint-of-aim. For example, if the point-of-aim lies on projection region60, distance measuring device 206 may bounce light off of projectionregion 60. In doing so, use may for instance be made of light-beamprojection device 202. Any other appropriate mechanism for determiningdistance may also be used for distance measuring device 206. The opticalaxes of light-beam projection device 202 and distance measuring device206 may coincide, for instance by making use of partially-reflectingelements (not shown) such as disclosed in U.S. Pat. No. 4,768,028 toBlackie (1988).

The operation of the present embodiment will now be described withreference to FIGS. 2, 10, and 3. The process elements in FIG. 3 will beidentical to the ones described in the first embodiment, resulting insimilar assumptions regarding the shape of interaction region 71 (andinteraction structure 72) and possibly orientation and position ofinteraction structure 72. In particular, a default distance betweeninteraction structure 72 and the origin of coordinate system x y z willbe assumed for purpose of initialization.

Referring to FIG. 11, instead of the program elements described abovewith reference to FIG. 4, program flow now continues at 120 a. Programelements 120 a-120 h are shown to be similar to program elements 90 a-90m (FIG. 4), except that only point CA, set A and counter a areconsidered. That is to say, in the present embodiment it is no longernecessary to determine the second point CB and associated repositioningof pointing device 20 during the calibration procedure.

After program element 120 h, program flow continues with programelements outlined in FIG. 12. Referring to FIG. 12, program elements 130a-130 g are seen to be similar to steps 100 a-100 p (FIG. 5). Comparingprogram element 130 f with program element 100 f (FIG. 5), it can beseen that at 130 f the program also stores information on thepoint-of-aim-distance (211 in FIG. 2) between the origin of the x′ y′ z′coordinate system and at least one of points C(p). Comparing elements130 g and 100 p (FIG. 5), it can be seen that thesepoint-of-aim-distances (211 in FIG. 2) are also used in determining the3D coordinates of the points in set P.

With complete information on the 3D coordinates of the points in set P,it is then possible to construct a 3D description of interactionstructure 72 (in FIG. 2). Therefore, program elements 110 a-110 k asdescribed above with reference to FIG. 6 may also be followed in thepresent embodiment.

Thus, methods and means are disclosed that afford a highly flexible andeasily deployable system for interacting with a presentation in adirect-pointing manner at a location not specifically equipped for sucha purpose. Moreover, when utilizing the second preferred embodiment itis often sufficient for the user to highlight various calibration points721 a, 721 b, . . . from one and the same position, making thecalibration procedure even more easy to follow.

Another embodiment will now be explained with reference to FIG. 13. FIG.13 shows another embodiment of the pointing device 20. Pointing device20 may be equipped with any or all of the elements mentioned in theother embodiments, such as described above with reference to FIG. 1 andFIG. 10. The present embodiment may include a base station 30 that hasassociated with it a coordinate system x y z. However, in the presentembodiment it can be assumed that the position of pointing device 20will remain substantially unchanged relative to interaction region 71over an extended period of time, and that the origin of coordinatesystem x y z may be assumed to be virtually anywhere instead of beingrelated to the base station 30. Furthermore, if any of the axes of the xy z system are chosen to be related to (locally) well-established,independent and stationary directions such as, for example, the Earth'smagnetic field and/or the Earth's gravitational field, the Earth itselfmay be interpreted as embodying base station 30. Under suchcircumstances there may not be a need for an artificial base station 30;coordinate sensing device 201 may then, for example, be embodied bydevices sensing the directions of the Earth magnetic and gravitationalfields, such as accelerometers and a compass (for example, model no.HMR3300 device as manufactured by Honeywell International Inc.,Morristown, N.J.), and communication and control device 204 may beconfigured to communicate directly with the computer (not shown).

The operation of the present embodiment will now be described withreference to FIGS. 8, 13, and 3. The remainder of the description of thepresent embodiment will, for purposes of explanation, include theassumption that any separately embodied base station 30 (FIGS. 8 and 13)is omitted entirely. It is also assumed that one of the axes of theorthogonal coordinate system x y z is substantially parallel to theEarth's gravity field (or has a defined relationship thereto), while thesecond and third axes of the coordinate system are in a fixed, knownrelationship to the at least one geographically defined axis. In thepresent embodiment it is assumed that the relative position of pointingdevice 20 with respect to interaction region 71 remains substantiallyunchanged, therefore the origin of coordinate system x y z will, forpurposes of explanation, be chosen to coincide with a fixed point in thepointing device 20. For the purpose of the present embodiment, it shouldbe understood that the three instantiations of pointing device 20 asshown in FIG. 8 substantially coincide.

It can be assumed in the present embodiment that a display system isarranged at the venue where a presentation is to be made, as acombination of a portable projection device 40 and projection surface60, for example a wall. It will furthermore be assumed that this displaysystem is connected, using appropriate means, to the computer (notshown) that generates the presentation imagery.

Upon arriving at the venue, the user connects pointing device 20 to thecomputer (not shown), for example via a USB connection (not shown), orusing a wireless relay device (not shown). Also, the computer mayrecognize the connection and start a program, part of which may becontained in communication and control device 204 or in control logiccontained in the wireless relay device itself (not shown).Alternatively, the user may be required to load the program into thecomputer manually via a CD drive, a floppy drive, memory stick or thelike (not shown). In any case, the program may initiate a calibrationroutine that has as its object establishing the shape, position, sizeand orientation of a well-defined interaction structure 72, relative tothe x y z coordinate system, wherein the interaction structure 72 isassumed to substantially coincide with a scaled and parallel version ofinteraction region 71. The flow of this program will be explained withreference to FIG. 3.

At 80 a the program is initiated. At 80 b defaults are entered forinteraction region 71. Specifically, interaction region 71 is assumed tobe a substantially parallel and scaled version of a well-definedinteraction structure 72. Moreover, the respective corners ofinteraction region 71 and interaction structure 72 are assumed tosubstantially lie on lines intersecting each other in the origin ofcoordinate system x y z. It should be noted that FIG. 8 shows aninteraction region 71 and an interaction structure 72 that are almostcoincident, but this is only meant for illustrative purposes and is nota limitation on the scope of the invention. Again referring to FIG. 3,at 80 b default values are also established for the orientation andposition of the interaction structure 72. For example, the defaultvalues may provide that interaction region 71 is substantially aparallel and scaled version of an interaction structure 72 that is aflat rectangle of which the two most vertical sides are substantiallyparallel to the Earth's gravitational field, and of which the two mosthorizontal sides are substantially perpendicular to the Earth'sgravitational field; moreover, the default values may provide that therespective corners of interaction region 71 and interaction structure 72substantially lie on lines intersecting each other in the origin ofcoordinate system x y z. The default values may furthermore provide thatthe position of interaction structure 72 is such that the distancebetween pointing device 20 and the lower left corner of interactionstructure 72 is equal to 1. Note that other values for the foregoingdistance may be equally valid in the present embodiment. Based on theforegoing default values, calibration points 721 a, 721 c, 721 d, . . .can define characteristic features of interaction region 71 and can alsodefine characteristic features of interaction structure 72.

At 80 c a decision is made whether the default values for interactionregion 71 and interaction structure 72 should be accepted or overriddenby the user. If the default values are to be accepted, the program flowcontinues to 80 j, the details of which are explained below withreference to FIG. 11. If the defaults are to be overridden, the programflow continues with elements 80 d-80 i, during which the user mayoverride any of the default settings. The user may be aided duringprogram elements 80 d-80 i by a library of predetermined shapes,predetermined orientations and/or predetermined positions.Alternatively, the user can be provided with the capability to constructcustom shapes, orientations and/or positions. In any event, program flowcontinues to 80 j.

Referring to FIG. 11, program flow now continues to 120 a. Programelements 120 a-120 h are substantially similar to program elements 90a-90 m (FIG. 4), except that only point CA, set A and counter a. areconsidered. In the present embodiment it is not required to determine asecond point CB and an associated repositioning of pointing device 20during the calibration procedure.

After 120 h, program flow continues with elements described withreference to FIG. 14. In FIG. 14, program elements 140 a-140 g are shownto be similar to steps 130 a-130 g (FIG. 12). Comparing element 140 fwith element 130 f (FIG. 12), it can be seen that element 140 f does notinclude storing information for any directly measured distance. Element140 f need not include storing information on the position of thez′-axis, because the position of pointing device 20 is assumed to remainsubstantially unchanged over an extended period of time. Comparingelements 140 g and 130 g, it can be seen that any directly measureddistance is not needed to determine the 3D coordinates of the points inset P (see FIG. 4). In fact, having the default distance (set to 1)between pointing device 20 and the lower left corner of interactionstructure 72 is sufficient for operation of the system in the presentembodiment. To explain this fact, reference is again made to FIG. 8.Using the above default values, interaction structure 72 will becompletely defined by, for example, its upper-left corner C(1), itsupper-right corner C(2) and its lower-left corner C(3)

$\begin{matrix}{{C\; (1)} = {\lambda_{1} \cdot \begin{pmatrix}{{Rx}\; 1} \\{{Ry}\; 1} \\{{Rz}\; 1}\end{pmatrix}}} & (8) \\{{C(2)} = {\lambda_{2} \cdot \begin{pmatrix}{{Rx}\; 2} \\{{Ry}\; 2} \\{{Rz}\; 2}\end{pmatrix}}} & (9) \\{{C(3)} = {\lambda_{3} \cdot \begin{pmatrix}{{Rx}\; 3} \\{{Ry}\; 3} \\{{Rz}\; 3}\end{pmatrix}}} & (10)\end{matrix}$

lying on lines A(1), A(2) and A(3) that connect the origin to points(Rx₁, Ry₁, Rz₁), (Rx₂, Ry₂, Rz₂) and (Rx₃, Ry₃, Rz₃) respectively. Usingthese definitions, the conditions described above determinerelationships between the various variables that can be written as

$\begin{matrix}{{\begin{bmatrix}{Rz}_{1} & {- {Rz}_{2}} & 0 \\{Rx}_{1} & 0 & {- {Rx}_{3}} \\{Ry}_{1} & 0 & {- {Ry}_{3}}\end{bmatrix}\begin{pmatrix}\lambda_{1} \\\lambda_{2} \\\lambda_{3}\end{pmatrix}} = \begin{pmatrix}0 \\0 \\0\end{pmatrix}} & (12)\end{matrix}$

which will have non-trivial solutions only if the determinant of thematrix equals zero. Then, it can be shown that any combination of (λ₁,λ₂, λ₃) that can be written in the form:

$\begin{matrix}{\begin{pmatrix}\lambda_{1} \\\lambda_{2} \\\lambda_{3}\end{pmatrix} = {\begin{pmatrix}{{Rz}\; {2/{Rz}}\; 1} \\1 \\{( {{Rz}\; {2 \cdot {Rx}}\; 1} )/( {R\; z\; {1 \cdot {Rx}}\; 3} )}\end{pmatrix} \cdot \alpha}} & (13)\end{matrix}$

where α is any arbitrary real number, will generate solutions forinteraction structure 72 that are parallel to each other. The assumptionthat the distance between pointing device 20 and the lower left cornerof interaction structure 72 is 1 will hence uniquely generate one ofthese solutions. To see that other assumptions regarding this distancedo not influence the operation of the present embodiment, reference ismade to FIG. 15. In FIG. 15, projections of two parallel solutions forinteraction structure 72 are shown, obtained for different values ofdistances CA-C(3). When it is assumed, for purposes of explanation, thatthese two solutions are parallel to the z-x plane, FIG. 15 may beinterpreted as a projection of the two solutions onto the x-y plane.These projections are shown as the thick line segments C(3)-C(2) andCC3-CC2. FIG. 15 also shows the projection of point CA (i.e., the originof coordinate system x y z), from where the lower left and upper rightcorners of interaction structure 72 were highlighted, and line-segmentsC(3)-CA and C(2)-CA which coincide with the (projections of) thehighlighting lines A(3) and A(2) (not denoted as such in FIG. 15).Finally, line-segment BB1-CA indicates (the projection of) a pointingline that is consistent with a point-of-aim BB1 on the one andpoint-of-aim BB2 on the other of the two solutions for interactionstructure 72. To show that BB1 and BB2 represent the same horizontalcoordinate when measured relative to the appropriate solution forinteraction structure 72, the lengths of line-segments C(3)-BB1,C(2)-BB1, CC3-BB2, CC2-BB2, BBl-CA and BB2-CA are represented by bb1,bb2, b1, b2, aa12 and a12, respectively. It is sufficient to show thatbb1/bb2=b1/b2. Consider the following equalities:

$\begin{matrix}{\frac{{bb}\; 1}{\sin \; {\beta 1}} = \frac{{aa}\; 12}{\sin \; \alpha \; 1}} & (14) \\{\frac{{bb}\; 2}{\sin \; \beta \; 2} = \frac{{aa}\; 12}{\sin \; \alpha \; 2}} & (15)\end{matrix}$

while, at the same time

$\begin{matrix}{\frac{b\; 1}{\sin \; \beta \; 1} = \frac{a\; 12}{\sin \; {\alpha 1}}} & (16) \\{\frac{b\; 2}{\sin \; \beta \; 2} = \frac{a\; 12}{\sin \; \alpha \; 2}} & (17)\end{matrix}$

From this it follows that

$\begin{matrix}{\frac{{aa}\; 12}{{bb}\; 1} = \frac{a\; 12}{b\; 1}} & (18) \\{\frac{{aa}\; 12}{{bb}\; 2} = \frac{a\; 12}{b\; 2}} & (19)\end{matrix}$

which, in turn, provides that

$\begin{matrix}{{{bb}\; 1\; \frac{a\; 12}{b\; 1}} = { {{bb}\; 2\frac{a\; 12}{b\; 2}}\Rightarrow\frac{{bb}\; 1}{{bb}\; 2}  = \frac{b\; 1}{b\; 2}}} & (20)\end{matrix}$

This implies that BB1 and BB2 represent the same horizontal coordinatewhen measured relative to the appropriate solution for interactionstructure 72. Note that, if FIG. 15 is interpreted as a projection ontothe z-y-plane, it can be concluded that BB1 and BB2 also represent thesame vertical coordinate when measured relative to the appropriatesolution of interaction structure 72. Therefore the initial assumptionthat the distance between C(3) and CA is equal to 1 does not influencethe operation of the third preferred embodiment.

It is theoretically possible that there are no non-zero solutions forthe parameters λ₁, λ₂ and λ₃, when the determinant referred to above inequation (12) is non-zero. Such may be the case, for example, due toerrors in measurements or in the assumed rectangular shape ofinteraction region 71. In such cases additional techniques may be usedto find an acceptable solution for interaction structure 72. Forexample, a minimization routine may be used to find a solution forinteraction structure 72 that minimizes a summation of the distancesbetween some of its corners (which, by assumption, lie on linesconnecting the origin of coordinate system x y z with the corners ofinteraction region 71) and the highlighting lines.

With complete information on the 3D coordinates of C(1), C(2) and C(3)it is possible to construct a 3D description of interaction structure72. Therefore, program elements 110 a-110 k as described in the firstembodiment and explained above with reference to FIG. 6 may also befollowed in the present embodiment.

There may be situations in which the user is not able to hold pointingdevice 20 continuously at precisely the same position while performingthe actions required by the calibration procedure described in FIGS. 3,11 and 14, or during the use of the system as a direct-pointing device.Such changes in position of the pointing device 20 can cause errors inthe calculation of the point-of-aim. It will be appreciated that if thediscrepancy between the calculated and the true point-of-aim is smallenough, the visual feedback provided by the projection of the cursor(501 in FIG. 2; projected cursor not explicitly shown) during use of thesystem as a direct-pointing device will still afford the user theperception that direct-pointing is being performed.

There are many other methods capable of establishing position, size andorientation of interaction structure 72, as will be appreciated by thoseskilled in the art. For example, if the distance between pointing device20 and interaction structure 72 is presumed known, if interactionstructure 72 is assumed to be rectangular with two vertical and twohorizontal sides and if its aspect ratio (the ratio between itshorizontal size and its vertical size) is also presumed known, thenknowledge of two lines from CA to the upper-right and lower-left corneris sufficient to narrow the number of solutions for interactionstructure 72 down to 2, dictated by 2 solutions of a quadratic equation(see also paragraph 181). One further assumption is then required todetermine which of these 2 solutions is the correct one; this assumptionmay be in the form of a third line passing through yet anothercharacteristic point of interaction structure 72, but it may also be inthe form of knowledge of the (approximate) angle between the plane inwhich interaction structure 72 lies and a line connecting the origin toone of its corners. Other scenarios may also be conceived for whichsolutions may be devised that are within the scope of the generalmethods set forth herein.

In the present embodiment, using well-established and stationarydirections such as, for example, the Earth's magnetic field and theEarth's gravitational field, it may be possible to omit a separatelyembodied base station, making the present embodiment of the systempossibly more compact, less expensive to make and easier to deploy.

While the above description contains many specificities, these shouldnot be construed as limitations on the scope of the invention, Manyother variations are possible. For example, pointing device 20 may behand held, but may also be carried by the user in a different manner,such as by means of a headset, finger-worn ring or the like.

Although the first, second and third embodiments make use of theassumption that the interaction structure 72 and interaction region 71are quadrangles, this need not be the case even when the correspondingcomputer screen interaction region 51 is rectangular in shape. Forexample, projection region 60 may be spherical in shape. This may causethe projection of a square computer screen interaction region 51 toresult in interaction region 71 having the general shape of asphere-segment. Other shapes are also possible, depending on a number offactors such as the angle between the optical axis of projection device40 and projection region 60. It will be appreciated by those skilled inthe art that set P (see FIG. 4, step 90 a and FIG. 11, step 200 a) andmethod M (see FIG. 6, step 110 b) may become more complicated underthese circumstances, but may still be well defined. In fact, method Mand sets P, A, B may be constructed in any way that is advantageous tothe particular system application. It is also possible to determine morepoints and associated line sets than just CA and CB, and associated linesets A and B. Specifically, set P may have more or fewer than the numberof points described with reference to the foregoing embodiments,depending on the complexity of the presentation venue. Also, sets A andB may contain more lines than the minimum number needed to establish thecoordinates of the points in set P.

The present invention may make use of mathematical techniques notdescribed explicitly herein but well known to those skilled in the artto aid with various aspects of the present invention, for example in thedetermination of position and/or orientation of interaction structure72. The second example of the first embodiment, for example, wasdescribed as relying on three lines highlighting corners of theinteraction structure 72. It is also possible that a more accuratesolution for the position and/or orientation of interaction structure 72may be obtained when the user is requested to also highlight the fourthcorner of the screen. An appropriate minimization routine may then beused to find a solution characterized by, for example, a minimumaccumulated distance between the four corners of interaction structure72 and the closest highlighting line.

Moreover, although set P is described as containing points that areneeded to completely define interaction structure 72, other embodimentsof the invention may use so-called “control points” for quality controlpurposes. Such control points may be used to test the validity of theassumptions made concerning, for example, the shape, the position, sizeand/or orientation of interaction region 71 and, therefore, interactionstructure 72, as well as to test the accuracy of the measurements madeto establish the 3D features of the various lines. As an example,consider the case in which interaction structure 72 is taken to be arectangle for which all data needed to uniquely establish its size,orientation and position with respect to the x y z coordinate system hasbeen ascertained by the program elements described with respect to theforegoing embodiments. Moreover, in this example interaction region 71and, therefore, interaction structure 72, are assumed to substantiallycoincide with the projection of a rectangular computer screeninteraction region 51. Then, the computer may display a suitable screenmark 521 a, 521 b, . . . at, for example, the center of computer screeninteraction region 51, which may be projected by projection device 40onto projection region 60. It will be appreciated that the processelements described herein may be used to calculate 3D coordinates ofthis projected point, based on the available data regarding shape, size,orientation and position of interaction structure 72. The user may thenbe required to highlight the projected point using light-beam projectiondevice 202 and indicate the success of this action to the computer by,for example, activating button 203 a. If all measurements andhighlighting actions were accurate enough and no false assumptions weremade, the 3D coordinates of the projected point should substantiallyconform to the 3D characteristics of the z′-axis. If the discrepancy isdeemed too large the user may be required to repeat part or all of thecalibration steps outlined in the present invention.

There may also be more than one method M and sets P, A and B, each ofwhich may be associated with a different interaction structure 72 andinteraction region 71. There may be more than one interaction structure72 and interaction region 71, each of which may, but need not, lie onprojection region 60 and each of which may, but need not, be associatedby means of projecting screen marks 521 a, 521 b, . . . , with one ormore computer screen interaction regions 51.

Although coordinate sensing devices 201 and 301 (see for example FIG. 1)were in some embodiments described as being able to provide informationon the 3D position and 3D orientation of pointing device 20 (see forexample FIG. 2) relative to the x y z coordinate system, it should beunderstood that such information provision requirements may be relaxedunder some circumstances. Specifically, in the first embodiment it isonly required that the 3D position and 3D orientation of pointing line21 (see for example FIG. 2) be known, instead of having completeinformation on the position of pointing device 20 along the pointingline 21. That is, in some cases pointing device 20 may be moved alongpointing line 21 without loss of functionality. In such embodiments thecoordinate sensing devices need only provide information on the 3Dposition and 3D orientation of a line (as opposed to a line-segment)substantially intersecting pointing device 20. If coordinate sensingdevices are able to provide complete 3D information (i.e., 3D positionand 3D orientation) of pointing device 20, as opposed to pointing line21, the extra positional information may, for example, be used toimplement the capability to zoom in on a region around point-of-aim 210(not explicitly shown in FIG. 2); other actions may also be based on theextra positional information. The foregoing capability may be inferredfrom the description of the third embodiment, in which the coordinatesensing device is only used to provide information on the angle betweenthe pointing line and two other fixed lines, instead of full positionand orientation information about a line segment.

Although the first, second and third embodiments describe theapplication of the invention as a cursor control device, the inventionmay also be used to facilitate non-cursor-related applications. Suchapplications may include “virtual writing” on the interaction region 71,target practice and the like. Also, the invention may be used other thanfor presentations such as, for example, to control a cursor on atelevision screen in order to make selections from menus associated withsuch things as Digital Satellite Television; other applications may alsobe anticipated.

Moreover, the features of the present invention that enable the trackingof a point-of-aim relative to art interaction region may be enhanced byalgorithms that allow the filtering of involuntary, fast and/or smallhand-movements (that may be caused, for example, by the activation ofbuttons). Such algorithms may be used to control a point that moves lesserratically than the actual point-of-aim while still being associatedwith it. Such motion filtering may produce a more steady motion ofcursor 501. Such algorithms may also be used when the user is requiredto highlight calibration points 721 a, 721 b, . . . . Filter algorithmsare well-known in the art.

The present invention also contemplates the inclusion into pointingdevice 20 of sensors capable of detecting motion or acceleration, bothlinear and angular, in addition to or as part of coordinate sensingdevice 201. Such sensors may be used to detect unintended motion oracceleration of pointing device 20 that may be caused, for example, bytrembling of the user's hand. The programs described above withreference to FIGS. 5, 6, 12 and 14 may be enhanced by algorithms thatcompare the output of such sensors and/or coordinate sensing device topredefined thresholds, so that a change in measured orientation and/orposition by coordinate sensing device 201 may be recognized assignificant only if such output exceeds these thresholds. Usingthreshold selection, a change in cursor position may only be affected ifinferred to be intentional, avoiding the type of “jitter” associatedwith the use of regular laser pointers by a user with an unsteady hand.

Furthermore, pointing device 20 may be equipped with an image capturingdevice, such as a digital camera, preferably with zoom and controllablefocus capabilities. Other image capturing devices, in combination withappropriate optical devices, may also be used. As stated before, such adigital camera may be used as an embodiment of distance measuring device206. Referring to FIG. 16, such a digital camera or the like (not shown)may also be used to aid in the process of highlighting the calibrationpoints 721 a, 721 b, . . . . Images of calibration points 721 a, 721 b,. . . and light spots at point-of-aim 210 may be captured during thetime when the user is engaged in the highlighting process. These imagesmay be temporarily stored, together with the position and orientation ofthe z′-axis at the time the image was acquired. During the highlightingprocess, it is likely that point-of-aim 210 changes slightly. Theresulting sequence of images may then be used to more accuratelyestimate the orientation and position of the pointing line 21 thatconnects the origin of the x′ y′ z′ system to calibration point 721 a,721 b, . . . . For example, an averaging algorithm may be used.Alternatively, the stored position and orientation of the z′-axiscorresponding to the captured image for which the light spot atpoint-of-aim 210 is closest to the center of calibration point 721 a,721 b, . . . may be taken as the position and orientation of thepointing line 21 that connects the origin of the x′ y′ z′ system tocalibration point 721 a, 721 b, . . . .

Some features may also be omitted without affecting the overallapplicability of the present invention. Referring to FIG. 1, forexample, level-sensing device 303 and visible markings 302 may behelpful but are not of critical importance and may be omitted. Iflevel-sensing device 303 is present, the output may also be utilized tocorrect the orientation of the x y z coordinate system to ensure thatits x-y plane is substantially horizontal, as previously explained. Insuch a case, the visible markings 302 may no longer precisely indicatethe position of certain coordinate planes or even the origin of the x yz coordinate system. If the orientation of the uncorrected x-y-planewith respect to a horizontal surface is small enough, though, visiblemarkings 302 may still provide sufficiently reliable information oncoordinate planes and origin of the corrected x y z coordinate system.

Furthermore, the programs outlined in FIGS. 3, 4, 5, 6, 11, 12 and 14may be changed appropriately without loss of functionality, particularlywith respect to the order of some of the elements. The programs may alsobe written in any language that is advantageous to particular theimplementation, including without limitation C, Java, and VisualBasic.

Other embodiments that include automatic activation and de-activation oflight-beam projection device 202, or include provision for instructingthe cursor control routines to show or hide computer cursor 501 are alsowithin the scope of the present invention. For example, automaticactivation and de-activation of light-beam projection device 202 mayoccur depending on whether or not the z′-axis intersects one of theinteraction structures 72, or regions of space close to them. This maybe performed by instructions to the cursor control routines to show orhide computer cursor 501. Also, means may be provided to activatelight-beam projection device 202 manually.

Furthermore, pointing device 20 may include a conventional, indirectpointing device such as a trackball or the like, to be used insituations where direct pointing is not possible or not desired. As apractical matter, coordinate sensing device 201, in some embodiments incombination with coordinate sensing device 301, may be used to effectcursor control in an indirect manner if the intended use makes thisdesirable. To effect cursor control in indirect pointing applications,pointing device 20 may include a user input device that enables the userto select whether indirect cursor motion control is desired, or mayinclude sensors and/or algorithms that enable determining when directpointing actions are not possible. For example, an algorithm or sensormay be included that enables determination of whether pointing device 20is within the operational range of base station 30. The third describedembodiment, specifically, may be enhanced with algorithms and/orsensors, such as accelerometers, that enable to determination of whetherpointing device 20 has been displaced significantly from the position atwhich the calibration procedure was performed, making direct pointingactions no longer possible without additional calibration steps. As partof the third described embodiment, additional input devices andalgorithms may be included that enable the user to mark an orientationin space such that indirect cursor motion control may be effected basedon the changes in orientation with respect to this marked orientation,as described in the cited prior art. Other embodiments of a system mayalso be enhanced by such functionality and input devices.

In addition to the previously described methods to determine size,position and orientation of interaction region 71, other methods arealso within the scope of the invention. These include methods based onthe use of digital cameras and the like. The position and orientation ofsuch cameras, relative to the x y z coordinate system, may be tracked byusing a device such as coordinate sensing device 201 and 301. Forexample, given sufficient knowledge of the optical characteristics of adigital camera, 3D features of an object may be determined from one ormore images taken from one or more positions. Using such images, acomplete 3D description of interaction region 71 may be determined.

Another alternative for establishing JD position, size and orientationof interaction region 71 is provided by a digital camera or the likeused in addition to, or as embodiment of, the distance measuring device206. In such embodiments the direction relative to the x′ y′ z′coordinate system of the axis along which distance is measured may becontrolled and measured by mounting the digital camera and/or distancemeasuring device 206 in such a way that their orientation relative tothe x′ y′ z′ coordinate system may be controlled and measured. Otherimplementations in which the foregoing components are positionally fixedare also contemplated. For purposes of explanation, the axis along whichdistance is measured is denoted as the z″-axis, and the z″=0 position isassumed to coincide with the origin of the x′ y′ z′ coordinate system.The orientation of the z″-axis with respect to the x′ y′ z′ coordinatesystem, and also with respect to the x y z coordinate system maytherefore be assumed to be known. In this embodiment, calibration points721 a, 721 b, . . . may be displayed simultaneously, as projections ofscreen marks 521 a, 521 b, . . . , and may differ in appearance, suchthat they may be distinguished by image processing software.Alternatively, calibration points 721 a, 721 b, . . . may appear asprojections of screen marks 521 a, 521 b, . . . in an automaticallycontrolled sequence. The user may then be queried to direct pointingdevice 20 in the general direction of interaction region 71 in such away that the digital camera may image the calibration point underconsideration. An automated calibration sequence may then be executed,wherein the z″-axis is directed, in sequence, to calibration points 721a, 721 b, . . . etc. The image processing software may identify thevarious calibration points 721 a, 721 b, . . . and the 3D position ofthese points may then be determined from knowledge of the orientation ofthe z″-axis and the coordinates of the z″=0 position with respect to thex y z coordinate system, in addition to the measuredpoint-of-aim-distance 211 between the calibration points 721 a, 721 b, .. . and the origin of the x′ y′ z′ coordinate system. This embodimentmay also provide light-beam projection device 202 mounted in a way thatenables control of the direction of the light-beam relative to the x′ y′z′ coordinate system. This light-beam may be used to aid in positioningthe z″-axis. Embodiments where light-beam projection device 202 arefixed or left out entirely are also contemplated.

The present invention also contemplates situations in which parts of thecalibration procedures outlined may be repeated at appropriate times.For example, when the relative position of base station 30 andinteraction region 71 changes substantially there may be a need torecalibrate the system. Such may also be the case when the position withrespect to interaction region 71 of pointing device 20 changessignificantly, while operation of a particular embodiment of theinvention operated using the assumption that such position remainssubstantially unchanged. As another example, coordinate sensing device201 and/or coordinate sensing device 301 may include time-integratedacceleration measuring devices. In such a case accuracy of coordinatesensing may deteriorate over time, because of drift in the baseacceleration measurements. When the deterioration has become largeenough to be unacceptable to the user, the user may be queried to placepointing device 20 in certain defined positions with respect to the x yz coordinate system, so as to re-initialize coordinate sensing device201 and 301.

The present invention also contemplates the use of a plurality ofpointing devices 20. Each of the plurality of pointing devices 20 may beuniquely identifiable by the computer, for example by carrying a uniqueID code inside their respective communication and control device 204.Each of the plurality of pointing devices may be associated with aspecific cursor and/or a specific interaction region 71. For example, afirst pointing device may be used to control a corresponding cursor onthe left-hand one of two interaction regions 71, and a second pointingdevice may be used to control a corresponding cursor on the right-handone of two interaction regions 71. Alternatively, both pointing devicesmay be used to control a corresponding, identifiable cursor on the sameinteraction region 71.

Furthermore, the present invention contemplates the use of a pluralityof entities such as base station 30, a particular one of which may beassociated with the origin of the x y z coordinate system. Such a basestation may be designated as the ‘master base station’. Each of the basestations may include coordinate sensing device 201, so that theirrespective position and orientation relative to the master base stationcan be determined with greater accuracy than that afforded by thecoordinate sensing device 201 that is incorporated in pointing device20. Using such multiple base stations and coordinate sensing devices, 3Ddata pertaining to pointing device 20 may be determined relative to theclosest base station, and the relative position and orientation of thatbase station with respect to the master base station may be used toestablish the 3D data pertaining to pointing device 20 with respect tothe x y z coordinate system. This may be of particular use in asituation where a user desires to move to a position in the room notwithin practical range of the master base station. In such a case, theuser may take one of the additional base stations along. Since theposition and orientation of this base station with respect to the masterbase station may be established with higher accuracy, this base stationmay be viewed as a relay-station, effectively increasing the operationalrange of the entire system. Note that, in such a case, the position ofthis base station may be changed with respect to interaction structure72 without the need to re-execute the calibration procedures.Alternatively, signals from multiple base stations, as measured by thecoordinate sensing device 201 disposed in pointing device 20, may beused simultaneously to reduce measurement errors. Generally speaking,determining the 3D data pertaining to pointing device 20 with respect tothe x y z coordinate system by making use of only one base station 30may be less accurate than by making use of a plurality of base stations.Hence, the effective range of operation of pointing device 20 may beenhanced by distributing a plurality of base stations over a largeregion of space.

Referring to FIG. 17; in another embodiment, different devices thanlight-beam projection devices may be used to determine the position ofthe points in set P. For example, light-beam projection device 202 maybe omitted in an embodiment where pointing device 20 has the generalshape of an elongated body, such as a pointing stick. The length of thisstick may be predetermined or variable. For example, pointing device 20could be in the shape of an extendable, or telescopic pen such as theINFINITER Laser Baton sold by Bluesky Marketing—Unit 29, Six HarmonyRow, Glasgow, G51 3BA, United Kingdom. The process of highlighting acalibration point 721 a, 721 b, . . . , in this embodiment may bereplaced by the action of pointing to the calibration point 721 a, 721b, . . . , thereby guided by the elongated shape of pointing device 20instead of the light spot shown in FIG. 16 at point-of-aim 210. Distancemeasuring device 206 may also be provided in a related embodiment,similar to the second described embodiment. For example, distancemeasuring device 206 may include a device to measure the distance fromthe tip of the elongated body of pointing device 20 to the origin of thex′ y′ z′ coordinate system. In such an embodiment, in order to obtainmeasurements of point-of-aim-distance 211, as described for example atprogram element 130 f (see FIG. 12), the user might be queried to makephysical contact between the tip of pointing device 20 and thecalibration point. Referring to FIG. 17, pointing device 20 may alsocomprise contact-sensing device 207, for example at the tip of pointingdevice 20, capable of indicating physical contact between the tip ofpointing device 20 and a surface. Contact-sensing device 207 may be inthe form of a pressure-sensor or switch. Such sensors are known in theart.

The present invention also contemplates the use of various other sensorsintegrated in pointing device 20 and/or base station 30 to helpdetermine the position and/or orientation of interaction structure 72.For example, pointing device 20 and/or base station 30 may include(ultra)sonic emitting and detecting devices to enable measuringsubstantially the shortest distance between a known point in the x y zcoordinate system (e.g., the position of pointing device 20 or of basestation 30) and the plane in which projection region 60 lies. The usermay be queried to align pointing device 20 or base station 30 such thatthe distance measuring device is oriented substantially perpendicularlyto projection region 60. It should be understood that other types ofsensors may be included in pointing device 20 or base station 30 to helpconstrain the position and/or orientation of interaction structure 72.

It may be advantageous to put some parts disclosed in this applicationin a known relation to one another. For example, projection device 40and base station 30 may advantageously be incorporated in one and thesame physical device, or be in a fixed relationship to one another.Alternatively, projection device 40 may be equipped with a coordinatesensing device 201. Whatever means are used, the system may be set upsuch that the position and/or orientation of some components, such asprojection device 40 (and its components) with respect to coordinatesystem x y z is fixed, known, or can be measured. If additionalappropriate sensors are included (such as, for example, level sensingdevice 303, visible markers 302, etc.), even the position and/ororientation of moveable parts of projection device 40 with respect tocoordinate system x y z may be considered measurable and hence known. Inparticular, such an integrated device may furthermore incorporate adevice like pointing device 20, associated with a pointing line 21; thatis, it may be equipped with a device that allows the orientation and/orposition with respect to coordinate system x y z of a pointing line 21to be known, measured and/or adjusted (potentially automatically). Thepreferred embodiment that results is depicted in FIG. 19 where, byexample, the pointing line associated with integrated pointing device 20has been made to coincide with the optical axis 401 of projection device40. In this application, optical axis 401 is defined as the linesubstantially connecting the optical origin 402 of the projection beam(see also FIG. 21) with the projected center of computer screen image50, as projected by projection device 40. (Note that the projectedcenter of computer screen image 50 may coincide with the center ofprojection image 70, notably so if both are rectangular.) Alternatively,the pointing line 21 may substantially coincide with another lineconnecting a characteristic point of projected image 70 (for example,one of its corners) to the optical origin 402 of projection device 40,or with other convenient lines. An even more advantageous embodiment mayresult when the pointing device 20 that is integrated with base station30 and projection device 40 also incorporates distance measuringdevice/image capture device 206, as indicated by the appearance ofpoint-of-aim-distance 211 in FIG. 19. To explain, it will now be assumedthat the origin of the x y z system coincides with the origin of the x′y′ z′ system of the pointing device 20 that is integrated with basestation 30 and projection device 40, as well as with the optical origin402 of the projection beam of projection device 40. It will also beassumed that the aspect ratio is known of a substantially rectangularcomputer screen image 50 that is to be projected by projection device40. If one furthermore assumes that projection device 40 is aligned insuch a way that its optical axis 401 is substantially perpendicular to(a flat) projection region 60 and that its optical characteristics (suchas the dimensions of its projection beam and the size of theto-be-projected image) are known, it will be clear to those skilled inthe art that knowledge of the distance between the optical origin ofprojection device 40 and (the center of) projection image 70 issufficient to uniquely establish the position, shape, orientation andsize of projection image 70 (and interaction region 71 and interactionstructure 72, assumed in this example to substantially coincide withprojection image 70), as measured with respect to the x y z coordinatesystem. Notably, in such a scenario there would not be a need for theuse of a user-wielded pointing device 20 to aid in establishing theabove parameters. Indeed, under these circumstances a user-wieldedpointing device 20 may be used as a direct-pointing instrumentstraight-away, without user-assisted calibration, after the abovedescribed automated calibration procedure has been executed so as toestablish all necessary parameters of projection image 70 (andinteraction region 71 and interaction structure 72).

If the integrated device (formed by at least combining, in the fashiondescribed above, projection device 40 and base station 30, and possiblya pointing device 20) is not aligned perfectly perpendicular toprojection region 60, an effect commonly referred to as keystoning willresult in a projection image 70 that is not rectangular. For example, ifthe optical axis 401 is perpendicular to (a vertically oriented)projection region 60 in a horizontal but not in a vertical sense,projection of a rectangular computer screen image 50 will generallyresult in a projection image 70 that is an isosceles trapezoid with twohorizontal edges. This is the most common form of image distortion, andis known as vertical keystoning. Conversely, if the optical axis 401 isperpendicular in a vertical, but not in a horizontal sense, theresulting isosceles trapezoid will have two vertical edges; this effectis known as horizontal keystoning. If both horizontal and

vertical misalignments are not too severe, the assumption thatprojection image 70 (and interaction region 71 and interaction structure72) is rectangular may still result in a system that affords the feelingof direct pointing, even if the system also assumes the vertical andhorizontal alignments are perfectly perpendicular; the reasons for thisare explained in paragraph 95.

In the more general case, as depicted in prior art FIG. 20, one or bothof these misalignments may be substantial. In such a case, the keystoneeffect will result in a projection image 70 that is neither rectangularnor shaped like an isosceles trapezoid. As explained in numerous priorart (such as U.S. Pat. No. 6,520,647 to Mitsubishi Electric ResearchLaboratories Inc., and U.S. Pat. No. 6,877,863 to Silicon Optix Inc.),vertical keystoning can easily be corrected for by incorporating sensorslike level sensing device 303—see also prior art FIG. 20. Using suchsensors, the vertical angle between the (vertically oriented) projectionimage 70 and optical axis 401 can easily be established. If,furthermore, distance measuring device 206 is subsequently used tosubstantially establish the distance between the projected version ofthe center of computer screen image 50 and the optical origin 402 ofprojection device 40 (i.e., measured along the optical axis 401), thoseskilled in the art will appreciate that inclusion of information on theoptical characteristics of projection device 40 will, once again,provide sufficient information to be able to establish position, shape,orientation and size of an interaction structure 72 that isnear-coincident with an interaction region 71 and projection image 70,provided the horizontal keystone effect is not too severe. Note thatthis may be done independent of whether or not the vertical keystoneeffect is corrected for.

If the horizontal keystone effect is also severe, the above cited priorart teaches that additional sensors may be used to establish thehorizontal angle between projection region 60 and optical axis 401, andthat they may be used to correct for the horizontal keystone effect.Using both vertical and horizontal keystone correction, a situation asdepicted in prior art FIG. 20 may thus arise, characterized by anon-perpendicular alignment both in horizontal and vertical sense,nonetheless resulting in a substantially undistorted rectangularprojection image 70. Such sensors may be devices such as used indistance measuring device/image capturing device 206—see also prior artFIG. 20. Alternatively, distance measuring device 206 may be used toestablish the distance from the optical origin 402 of the projectionbeam to two well-chosen characteristic points of projection image 70.One characteristic point might be the projected center of computerscreen image 50; the other one might be the projected version of a pointhalf-way the left vertical edge of computer screen image 50. Simpleapplication of the cosine rule (see FIG. 21, depicting a top-view ofFIG. 19 for a case where the horizontal alignment is not perfectlyperpendicular) shows that, if distances b and c and angle α are known(the latter angle following from knowledge of the optical dimensions ofthe projection beam), all other quantities in the triangle can becalculated—including the sought-after angle β. Given the above, it willbe appreciated that inclusion of any such sensors will allow a uniquedetermination of position, shape, orientation and size of projectionimage 70 (and a near-coincident interaction region 71 and interactionstructure 72), even in the general case of a projection device 40(integrated in the above described sense with at least base station 30and possibly a pointing device 20) that is misaligned with respect toprojection region 60 both in the horizontal and vertical sense. Noteagain that this may be done independent of whether or not the verticaland/or horizontal keystone effect is corrected for.

Finally, then, it will be understood that use of an image capturingdevice 206 of which the position and orientation in the x y z coordinatesystem is known may be of great help in the presently discussedembodiment, as already alluded to in paragraph 158 and 159. Indeed,those skilled in the art will recognize that distance measuring device206 may be embodied by combining knowledge of the opticalcharacteristics of projection device 40 (such as the dimensions of itsprojection beam and the size of the to-be-projected image) withinformation obtained from a captured image of projection image 70 by asensor like image capture device 206. After all, if the size ofprojection image 70 (for ease of explanation, but not out of necessity,assumed to be rectangular) can be established using an image capturedevice 206, one may use knowledge of the optical characteristics ofprojection device 40 to easily establish the distance from its opticalorigin 402 to, for example, the projected center of computer screenimage 50. This way, then, 3D parameters describing projection image 70,interaction region 71 and interaction structure 72 can easily bedetermined, as will be appreciated by those skilled in the art.Alternatively, focus mechanisms in image capture device 206 may also befound useful in this regard. In short, it will be understood that anydevice capable of establishing the distance from a known point in the xy z coordinate system (such as, for example, the optical origin 402 ofthe projection beam of projection device 40) to a point with apredetermined relation to projection image 70, interaction region 71 orinteraction structure 72 is contemplated by the current application. Itwill also be understood that, although the above continuously referredto the projected center of computer screen image 50, this was merelyintended as an example and such a point may, without loss offunctionality, be replaced by any point related in a known fashion tocomputer screen image 50, projection image 70, interaction region 71 orinteraction structure 72.

Furthermore, other techniques disclosed in this application may becombined with the above described methods, including but not limited tothose outlined in paragraphs 149, 155, 160, 161, 162, 163 and 164. Inparticular, calibration sensors associated with projection device 40 orbase station 30 or an integrated pointing device 20 and their associatedcalibration methods may be combined with methods related to auser-wielded pointing device 20, as disclosed in this application. Forexample, the system may require the user to highlight one or morecalibration marks 721 a,b, . . . with a user-wielded pointing device 20,in order to correct for the horizontal non-alignment of projectiondevice 40 and projection region 60 (the horizontal keystone-effect),while sensors incorporated in projection device 40 or base station 30may be used to automatically correct for any vertical misalignment.Also, control points (see also paragraph 14 7) may be used to check thevalidity of the calculated position, orientation, shape and size ofprojection image 70, interaction region 71 and/or a near-coincidentinteraction structure 72. Also, it will be understood that in certainembodiments base station 30 may be left out entirely. Furthermore,distance measuring device 206 may also advantageously be left out. Toexplain, the reader is referred to the embodiment described inparagraphs starting at paragraph 126. If the methods and devicesdescribed there are combined with the methods and devices outlined inparagraphs 167-171, another advantageous embodiment will result. That isto say, if the position of a user-wielded pointing device 20 is not toodifferent from the optical origin 402 of the projection beam ofprojection device 40, such that they may be considered to substantiallycoincide, it will be appreciated by those skilled in the art that asystem results that does not have a need for position sensing, anassociated base station 30 or distance measuring device 206.Particularly when projection region 60 is sufficiently far away fromprojection device 40, such that projection image 70 is relatively large,any small discrepancy between the position of a user-wielded pointingdevice 20 and the position of projection device 40 will be negligible.In such a case, knowledge of the dimensions of the projection beam(assuming that the angle between its optical axis 401 and projectionregion 60 is substantially known) will be sufficient to substantiallyascertain the point-of-aim of user-wielded pointing device 20, as willbe obvious from the methods disclosed in this application. Animprovement on the above system may be obtained by including one or moredistance sensing devices 206 that may, for example, be used to establishthe horizontal and/or vertical angles between optical axis 401 andprojection region 60, as described above. This way, it is possible tocompensate for a non-perpendicular alignment that might, withoutinclusion of such sensors, cause intolerable errors in the computationof the point-of-aim of a user-wielded pointing device 20.

Yet another preferred embodiment results as a variation on a combinationof the embodiments described in paragraphs 120-125 and paragraphs126-145, and will be disclosed here with reference to FIGS. 3, 11, 12,22, and 23. It will be understood that, for reasons of clarity,projection device 40 has not been drawn in FIG. 23. The presentembodiment is characterized by a pointing device 20 that is equippedwith at least a distance measuring device 206 and a coordinate sensingdevice 201 that may be limited to sensing orientation. Such a coordinatesensing device 201 may for example rely on an electronic compass likethe HMR3300, on gyroscopes, on combinations of the above, or on anyother suitable technology. It will be assumed that the position ofpointing device 20 does not change substantially while the calibrationprocedures described in FIGS. 3, 11 and 12 are executed, as indicated inFIG. 23 by the two near-coinciding instantiations of pointing devices20. For purpose of explanation it will therefore be assumed that theorigin of coordinate system x y z and the origin of coordinate system x′y′ z′ are coincident, at least while the above-mentioned calibrationprocedures are executed. Finally, it will be assumed that one of theaxes of the orthogonal coordinate system x y z is substantially verticalwhile the other two are substantially horizontal (the inclusion of alevel-sensing device 303, not specifically shown in FIG. 22 for reasonsof clarity, may be beneficial in this context).

The operation of the present embodiment will now be described withreference to FIGS. 3, 11, and 12. The process elements in FIG. 3 will beidentical to the ones described in the first embodiment, resulting insimilar assumptions regarding the shape of interaction region 71 (andinteraction structure 72) and possibly orientation and position ofinteraction structure 72. In particular, a default distance betweeninteraction structure 72 and the origin of coordinate system x y z willbe assumed for purpose of initialization.

With reference to FIG. 11, program flow now continues at 120 a. Programelements 120 a-120 h are shown to be similar to program elements 90 a-90m (FIG. 4), except that only point CA, set A and counter a areconsidered. That is to say, in the present embodiment it is no longernecessary to determine the second point CB and associated repositioningof pointing device 20 during the calibration procedure, since theposition of pointing device 20 is assumed not to change substantiallyfrom the origin of coordinate system x y z during execution of the stepsconsidered.

After program element 120 h, program flow continues with programelements outlined in FIG. 12. Referring to FIG. 12, program elements 130a-130 g are seen to be similar to steps 100 a-100 p of FIG. 5. Comparingprogram element 130 f with program element 100 f (FIG. 5), it can beseen that at 130 f the program also stores information on thepoint-of-aim-distance between the origin of the x′ y′ z′ coordinatesystem and at least one of points C(p), indicated in FIG. 23 as 721 d.Comparing elements 130 g and 100 p (FIG. 5), it can be seen that thesepoint-of-aim-distances (211 in FIG. 23) are also used in determining the3D coordinates of the points in set P.

With complete information on the 3D coordinates of the points in set P,it is then possible to construct a 3D description of interactionstructure 72 (see FIG. 23). Therefore, program elements 110 a-110 k asdescribed earlier with reference to FIG. 6 may also be followed in thepresent embodiment.

As in previously a described embodiment, the above outlined embodimentmay be satisfactorily used as a direct-pointing system if the positionof pointing device 20 does not change substantially from the origin ofcoordinate system x y z. When the user moves to a different part of theroom the position of pointing device 20 will, by assumption, be unknown(since coordinate sensing device 201 was assumed to only be capable ofsensing orientation). This position then needs to be reestablished inorder to be able to execute steps like 110 f and 110 h (see FIG. 6),which allow the system to be used for direct-pointing cursor control.

In order to reestablish the position of pointing device 20 the user maybe required to direct pointing line 21 to substantially pass through apoint of which the 3D coordinates are substantially known in coordinatesystem x y z. In doing so, the user may but need not be assisted bylight-beam projection device 202. This point could, for example, be acorner point or any other well-defined point of interaction structure 72(which, by assumption, coincides with interaction region 71).Advantageously, it could be the last position that projected cursor 701(see FIG. 23) was directed to. After all, the 3D characteristics ofinteraction structure 72 (and interaction region 71) are known, as isthe position of computer cursor 501; the inverse of map M can theneasily be used to compute the 3D characteristics of projected cursor701. It will also be apparent that measurements of the orientation ofpointing line 21 and point-of-aim-distance 211 will then constituteenough information to calculate the position of pointing device 20 incoordinate system x y z. With the position of pointing device 20reestablished, the present embodiment can again successfully be used tocontrol the computer cursor in a direct-pointing manner. This willremain to be the case as long as the position of pointing device 20 doesnot stray far from the recently reestablished position.

The applicant recognizes that the action of directing pointing line 21through the above-mentioned well-defined point (whilst measuringpoint-of-aim-distance 211 and orientation of pointing device 20) mayresult in a pointing line 21 not perfectly passing through this point,especially if the user is not aided by light-beam projection means 202.Indeed, any errors made during this action will result in similarmismatches between subsequently calculated points-of-aim and truepoints-of-aim of pointing device 20. These mismatches may not be tooapparent to the user though since, presumably, the user was not capableof discerning such a mismatch when attempting to direct pointing line 21through the above mentioned well-defined point, as will be appreciatedby those skilled in the art. Consequently, the present embodiment maystill afford the user the feeling of direct-pointing cursor controlunder such circumstances.

Sensors capable of sensing position may but need not be added to thecoordinate sensing device 201 of the current embodiment. Indeed, even ifthese sensors were only capable of maintaining positional accuracy overa limited period of time (e.g., sensors based on accelerometertechnologies), their inclusion could still be advantageous. For example,such sensors may be employed to accurately track the position ofpointing device 20 over the relatively short period of time that startswhen the user directs pointing line 21 through the above-mentionedwell-defined point (whilst measuring point-of-aim-distance 211 andorientation of pointing device 20), and ends when the user indicates hisor her desire to stop direct-pointing cursor control. This way,direct-pointing cursor control can still be maintained even if theposition of pointing device 20 strays relatively far from the positionit was in at the start of the above-mentioned period of time. In asimilar way, such position sensors may also be beneficial during the(relatively short) calibration procedures aimed at establishing the 3Dcharacteristics of interaction structure 72 (and, by assumption,interaction region 71), and can help to reduce errors that are due to aviolation of the assumption that the position of pointing device 20remains constant. The inclusion of such sensors may even obviate theneed for distance measuring device 206 during calibration, when they areused according to the methods of earlier described embodiments (see forexample paragraphs 107-113).

Distance measuring device 206 may be embodied by a laser sensor such asimplemented in the TruLaser TLM100 Distance Measurer manufactured byStanley. Such sensors are exceptionally well suited for the presentembodiment, being capable of accurately determining distance along apointing line 21 to a point on a plane, even if the angle between theplane's normal and pointing line 21 is large. Other distance sensorsthat experience more difficulty under such circumstances may howeveralso be used satisfactorily. For example, (ultra)sonic sensors such asthe MaxSonar-EZl as manufactured by Maxbotix have a measurement beamthat is far broader than that of the above mentioned laser-sensor. Such(ultra)sonic sensors are known to be susceptible to measurement errorswhen the beam's central axis is at a large angle to the normal of asmooth plane it is aimed at. Approximate knowledge of this angle mayconceivably be used to at least partially correct for such errors, aswill be appreciated by those skilled in the art.

A system based on coordinate sensing device 201 capable of sensingorientation (such as the HMR3300 electronic compass) in combination withan (ultra)sonic distance measuring device 206 (such as the MaxSonar-EZl)constitutes a particularly economical and consequently advantageouspreferred embodiment, which will now be described. Using methodsoutlined in the embodiment described in paragraphs 126-145, theorientation, size and position of an interaction structure 72 that is ascaled and parallel version of interaction region 71 can readily befound. For example, if it is assumed that interaction region 71 issubstantially a rectangle with horizontal and vertical sides, it issufficient to consider only 3 calibration lines. If it is furthermoreassumed that the aspect ratio of this rectangle is known, only 2calibration lines suffice to narrow the number of possible solutions forinteraction structure 72 down to 2. This can be understood byconsidering an example of two calibration lines passing through lowerleft corner C(3) and upper right corner C(2), depicted as 721 d and 721a in FIG. 23. If the lower left corner of interaction structure 72(which is assumed to be a scaled and parallel version of the rectangularinteraction region 71) lies at such a distance from the origin that itsz-coordinate equals 1, we find

$\begin{matrix}{{C\; (2)} = {\lambda_{2} \cdot \begin{pmatrix}{{Rx}\; 2} \\{{Ry}\; 2} \\{R\; z\; 2}\end{pmatrix}}} & (21) \\{{C(3)} = \begin{pmatrix}{LLx} \\{LLy} \\1\end{pmatrix}} & (22)\end{matrix}$

Here, LLx, LLy, Rx2, Ry2 and Rz2 are known, since C(3) and C(2) lie onpointing lines 21 of which the orientation has been measured. If wesubsequently require these points to define a rectangle with horizontaland vertical sides and aspect ratio rat, we find

√{square root over ((LLx−λ ₂ ·Rx2)²+(LLy−λ ₂ ·Ry2)²)}=rat·√{square rootover ((λ₂ ·Rz2−1)²)}⇒

(Ry2² +Rx2² −rat ² ·Rz2²)·λ₂ ²+

(−2·LLx·Rx2−2·LLy·Ry2+2·rat ² ≤Rz2)·λ₂+

(LLx ² −rat ² +LLy ²)=0  (23)

which can readily be solved for λ₂. Because of the quadratic naturethere will in general be 2 solutions to Equation (23), and additionaldata is needed to pick the correct solution. This may be in the form ofa third line passing through yet another characteristic point ofinteraction structure 72, but it may also be in the form of knowledge ofthe (approximate) angle between the plane in which interaction structure72 lies and a line connecting the origin to one of its characteristicpoints. The applicant realizes that the above equation may not alwaysgenerate real solutions for λ₂, since the pertinent determinant may benegative due to measurement errors, a mismatch between the true andassumed aspect ratio, etc. Those skilled in the art will appreciate,however, that there are many ways to find real values for λ₂ thatapproximately solve Equation (23). For example, the value for rat may beslightly varied from its a-priori value to the point where Equation (23)yields real solutions, which may be acceptable as solutions to theoriginal problem. Other ways of finding approximate solutions are alsocontemplated by the current application.

It will be understood that an interaction structure 72 thatsubstantially coincides with interaction region 71 can be readily foundif the distance from the origin to at least one of the characteristicpoints of the latter is measured. Such a measurement can be made usingaccurate sensors based on (for example) laser technology. If(ultra)sonic sensors are used for this purpose, knowledge of the anglebetween the (ultra)sonic beam (for purpose of explanation assumed to besubstantially centered on pointing line 21) and the normal to the planein which interaction region 71 lies may prove beneficial, as will beappreciated by those skilled in the art of (ultra)sonic distancemeasurement. This angle can be readily calculated, as outlined above,once the orientations of a minimum of two calibration lines have beenmeasured, and may advantageously be used during processing of any(ultra)sonic distance measurement(s).

The accuracy and reliability of the (ultra)sonic distance measurementmay be greatly enhanced if the angle between the (ultra)sonic beam andthe normal to the plane in which interaction region 71 lies is small.For this reason, the user may advantageously be asked to execute theentire calibration procedure from a position that meets such arequirement. It will be obvious that the fulfillment of this requirementmay be tested once the orientation of at least 2 calibration lines hasbeen measured. In general, the user may be directed to perform therequired distance measurement(s) whilst orienting pointing device 21 inone or more directions deemed most advantageous. For example, the usermay be asked to substantially point at the center of interaction region71, at certain points on its edges, or trace out a diagonal ofinteraction region 71 whilst multiple distance measurements are taken.Many other ways of measuring a suitable distance parameter are possible.With reference to FIG. 24 (for purpose of clarity projection device 40has been omitted from the drawing, and interaction region 72 andinteraction structure 71 have been depicted as coincident), it may insome cases be advantageous to take a distance measurement whilstdirecting pointing device 20 at a point that does not lie in interactionregion 71, but has a defined relationship to it. Such points may includea point 721 e that substantially lies in the same horizontal plane aspointing device 20 (that is, with vertical coordinate substantiallyequal to 0) as well as in the plane in which interaction region 71 lies(which is assumed vertical). This will result in a measurement for whichthe (ultra)sonic beam and the plane in which interaction region 71 liesare substantially perpendicular in the vertical sense, which mayconstitute more optimal conditions for the use of (ultra)sonicmeasurement devices. This can be advantageous in situations whereinteraction region 71 is mounted so high above pointing device 20 thatany pointing line 21 pointed at interaction region 71 will have a largeangle with the normal to the plane in which interaction region 71 lies,as depicted in FIG. 24. Even more optimal circumstances might be createdif point 721 is chosen such that pointing line 21 perpendicularlyintersects the plane in which interaction region 71 lies in bothhorizontal and vertical sense. Other points that have a definedrelationship to interaction region 71 may also be used advantageously.It will be evident that the validity of desirable conditions (such asthe (ultra)sonic beam being near-horizontal) can easily be checked,since the orientation of pointing device 20 with respect to coordinatesystem x y z, as well as with respect to interaction region 71, areknown, measurable or can be calculated.

Even without distance measuring device 206, the outlined methods providefor a direct-pointing cursor control system as long as pointing device20 substantially remains at one and the same position. When the userdecides to move to a different location in the room, depicted in FIG. 24as the third instantiation of pointing device 20, the position ofpointing device 20 will be unknown and has to be at least partiallyreestablished. As explained earlier, this may be accomplished byrequiring the user to direct pointing line 21 through a well-definedpoint of which the 3D coordinates are known in coordinate system x y z,whilst measuring the orientation of pointing line 21 andpoint-of-aim-distance 211. However, an (ultra)sonic distance measuringdevice 206 may not be capable of establishing point-of-aim-distance 211with sufficient accuracy, especially when the angle between pointingline 21 and interaction region 71 is unfavorable. Also, measuringpoint-of-aim-distance 211 may take too much time to be convenient duringoperation of the invention as a direct-pointing cursor control system.In such cases, with reference to FIG. 24, use may be made of the factthat an average user will tend to wield pointing device 20 in the samehorizontal plane as from which the initial calibration routine wasexecuted. Under such circumstances, the position of pointing device 20can be reestablished without the need for a distance measurement, namelyas the intersection of pointing line 21 with a horizontal plane withvertical coordinate z=0, i.e., with a plane through the origin ofcoordinate system x y z (chosen such that it contains the point fromwhere the calibration procedures were executed). Advantageously, thewell-defined point mentioned above may be chosen such that the anglebetween pointing line 21 and the normal to the plane with verticalcoordinate z=0 is as small as possible, so as to minimize the error inthe calculated position. Other choices are also possible; goodcandidates may include points close to the upper edge of interactionregion 71 such as its upper right corner 721 b, depicted in FIG. 24. Itwill be appreciated that small errors in calculated position resultingfrom the fact that pointing device 20 is not always held exactly at thesame vertical position may still afford the user the feeling that thecursor is controlled in a direct-pointing manner. To the user, then, theinvention affords a system where the cursor may be ‘grabbed out of thetop right screen corner’, and subsequently moved about in adirect-pointing manner.

Those skilled in the art will appreciate that the user may but need notbe required to ‘grab the cursor’ every time he or she desires to engagein direct-pointing cursor control. For example, pointing device 20 maybe equipped with two buttons that activate cursor control, namely onethat requires the user to first direct pointing line 21 through awell-defined point and another that presumes the position of pointingdevice 20 has not substantially changed since the last time the cursorwas activated. Alternatively, the user may be required to indicate hisor her desire to reestablish position in other ways, for example byexecuting a predefined motion pattern such as shaking pointing device20; coordinate sensing device 201, possibly in combination withadditional sensors such as accelerometers and gyros may be of help underthese circumstances. Still alternatively, the system may includealgorithms aimed at detecting when pointing device 20 may have strayedfar enough to warrant reestablishing its position in coordinate system xy z. Such detection algorithms may for example be based on elapsed time,on inertial measurements made by gyros and/or accelerometers that may bepart of coordinate sensing device 201, on distance measurement devicescapable of sensing received power from a transmitting device fixed inspace, etc. All of these and similar methods are contemplated by theinvention.

It will also be appreciated that inclusion of coordinate sensing devices201 that are capable of maintaining positional accuracy over a limitedamount of time (e.g., inertial sensors) may obviate the need fordistance measuring device 206 both during calibration as well as duringoperation of the present embodiment. This may be understood byconsidering calibration methods outlined in earlier describedembodiments (see for example paragraphs 107-113) m combination withmethods outlined in paragraph 184.

Thus, methods and means are disclosed that afford a compact system forinteracting with a presentation in a direct-pointing manner, whilstleaving the presenter free to move about the presentation venue. Thissystem, moreover, obviates the need for a base-station 30 that embodiesa coordinate system x y z, and for a coordinate sensing device 201capable of continuously tracking positional coordinates of pointingdevice 20 in this coordinate system.

It will be obvious that elements of the various disclosed embodimentscan advantageously be combined. In particular, combining the embodimentsdescribed in paragraphs 167-172 and paragraphs 173-188 will result in asystem that may calibrate itself automatically, and leaves the presenterfree to move about the presentation venue whilst engaging indirect-pointing interaction with the display. Moreover, the resultingsystem obviates the need for a base-station 30 that embodies acoordinate system x y z, and for a coordinate sensing device 201 capableof continuously tracking positional coordinates.

In particular, when use is made of distance measurement devices 206 forwhich the sensitive axis is relatively poorly defined (such as(ultra)sonic devices), methods disclosed in paragraphs 167-172 and182-188 may still result in an economical and versatile system for whichthe calibration procedure is almost completely automatic. To clarify,reference is made to FIG. 25, where projection device 40 is integratedwith a pointing device 20 that contains at least distance measurementdevice 206 and coordinate sensing device 201. As explained in previousembodiments, this may result in a system capable of automaticallyestablishing size, orientation and position of interaction structure 72(and, by assumption, interaction region 71) as measured in a coordinatesystem x y z that has a predetermined relation to projection device 40.Moreover, since the line-of-sight between projection device 40 andinteraction region 71 will in general be unimpeded, and since the anglespertinent to the problem are very well defined, those skilled in the artwill appreciate. that distance sensing device 206 may advantageously beembodied by (ultra)sonic sensors. What remains, then, is to find thevertical coordinate of the plane in which the user generally deploys theuser-wielded pointing device 20, depicted in FIG. 25 as the ‘z=constant’plane. To this end, the user-wielded pointing device 20 may also beequipped with a distance measuring device 206, and the user may berequired to perform a one-time calibration step by directing it towardsa known point in coordinate system x y z whilst measuring bothorientation of the pertinent pointing line 21 as well as the pertinentpoint-of-aim-distance 211. This known point can advantageously be chosento coincide with projection device 40, particularly so if distancesensing devices 206 of both projection device 40 and user-wieldedpointing device 20 are based on (ultra)sonic technology. In such a caseit will be obvious to those skilled in the art that the pertinentpoint-of-aim-distance 211 can be measured with relatively high accuracy,since (ultra)sonic transducers will be present both at the target and atthe source. Alternatively, the user may be asked to direct user-wieldedpointing device 20 to two or more known points in coordinate system x yz, such as top right and lower left corner of interaction region 71,after which its position can be found by triangulation. In the lattercase there is not even a need to incorporate a distance sensing device206 in user-wielded pointing device 20. Whatever choice is made, theabove outlined methods yield enough information to establish theposition of user-wielded pointing device 20 and, in particular, thez-coordinate of the plane in which it generally resides, so that methodsoutlined in paragraphs 182-188 may be successfully employed tore-establish its position in case it is moved to a different location.As such, this system will be seen to be highly flexible, relativelyinexpensive to manufacture, and extremely user-friendly. Indeed, thenecessary equipment may come in the form of modules that can beretrofitted to existing projection devices 40 not specifically equippedfor this application.

It will be understood that in some cases it may be unnecessary toprovide a separately embodied base station 30, whilst still providing acoordinate sensing device 201 capable of tracking orientational andpositional coordinates. For example, coordinate sensing device 201 maypartly or completely rely on inertial sensing means such asaccelerometers and gyroscopes, or on distance measurements with respectto walls, ceiling and/or floor. In such cases it would be possible toleave out base station 30 entirely and simply choose an appropriate x yz coordinate system. In such embodiments communication and controldevice 204 could be configured to communicate directly with thecomputer.

The present invention also contemplates the use of automated proceduresto establish the nature of the calibration points 721 a, 721 b, 721 c,721 d, based on the measurements of the orientation and/or position ofthe pointing lines 21 that substantially pass through them. For example,with reference to FIG. 7, calibration points 721 a and 721 d need not beexplicitly identified to the user as upper-right and lower-left cornerrespectively, but may merely be identified as diagonally oppositecorners. Those skilled in the art will appreciate that in such a case anautomated procedure may use the measurements of orientation and/orposition of both pointing lines 21, specifically with respect to thex-y-plane, to identify which of the two lines passes through the upperone of the two diagonally opposite corners. The practical consequence tothe user is that the user need not be concerned about which of the twocalibration points 721 a and 721 d is highlighted first. Similararguments can be used to construct automated procedures that determinewhether a pointing line 21 substantially passes through a left or aright corner of interaction region 71 (and, by assumption, ofinteraction structure 72). For example, the a-priori assumption may bemade that the clock-wise angle between a pointing line 21 through a leftcorner and a pointing line 21 through a right corner, measured parallelto the x-y-plane, should not exceed 180 degrees, as will be appreciatedby those skilled in the art.

Finally, although the methods disclosed by the present invention aredescribed in 3D, the invention may also be applied in 2D scenarios.

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of thisdisclosure, will appreciate that other embodiments can be devised ‘whichdo not depart from the scope of the invention as disclosed herein.Accordingly, the scope of the invention should be limited only by theattached claims.

What is claimed is:
 1. A non-transitory computer-readable medium ormedia storing computer-executable instructions for directing a computerto perform a method for controlling the contents of a computer-generatedimage in conjunction with an enclosure comprising a plurality of sensorsand configured to generate a pattern of light representing the image,the plurality of sensors comprising: an accelerometer; a gyro; an imagecapturing device; a magnetometer; and a sensing device that is sensitiveto a position of the enclosure and orientation of the enclosure; themethod comprising the steps of: receiving data that is dependent on anoutput of at least one of the plurality of sensors; and controlling thecontents of the computer-generated image based on the data.
 2. Thenon-transitory computer-readable medium or media according to claim 1,wherein the plurality of sensors further comprises a user input deviceconfigured to allow the user to manually provide two dimensional input,the user input device being insensitive to position and orientation ofthe enclosure.
 3. The non-transitory computer-readable medium or mediaaccording to claim 2, wherein the plurality of sensors further comprisesa pressure sensor.
 4. The non-transitory computer-readable medium ormedia according to claim 2, wherein the pattern of light is generated bya display for displaying said image, the display being part of theenclosure.
 5. The non-transitory computer-readable medium or mediaaccording to claim 2, wherein the enclosure further comprises a laserconfigured to project a light spot in a direction that lies at anon-zero angle with the optical axis of the image capturing device. 6.The non-transitory computer-readable medium or media according to claim5, wherein the laser is configured to project infrared light, and theenclosure contains a computer-readable identification code thatdistinguishes the enclosure among a plurality of like enclosures.
 7. Thenon-transitory computer-readable medium or media according to claim 1,wherein the sensing device comprises the accelerometer, the gyro, theimage capturing device and the magnetometer.
 8. An apparatus forgenerating first data and second data for use in controlling acomputer-generated image, the first data being dependent on a positionof a first device relative to a second device, the second data beingdependent on a position of the second device, the first deviceconfigured to be handheld and wielded in mid-air by a user, the seconddevice configured to be taken along by the user, the apparatuscomprising: a first sensing device for generating the first data, thefirst sensing device being sensitive to a position of the first devicerelative to the second device; and a second sensing device forgenerating the second data, the second sensing device being sensitive toa posit ion of the second device; wherein the first device contains atleast one element of the first sensing device.
 9. The apparatusaccording to claim 8, wherein the first data is dependent on anorientation of the first device, and the second data is dependent on anorientation of the second device.
 10. The apparatus according to claim9, wherein the first sensing device is sensitive to three independentpositional coordinates of the first device relative to the second deviceand three independent orientational coordinates of the first device, andthe second sensing device comprises a magnetometer and is sensitive tothree independent positional coordinates of the second device and threeindependent orientational coordinates of the second device.
 11. Theapparatus according to claim 10, wherein the first sensing devicecomprises a first accelerometer and a first gyro, and the second sensingdevice comprises a second accelerometer and a second gyro and issensitive to three independent positional coordinates of the seconddevice relative to a point external to the first device and to thesecond device.
 12. The apparatus according to claim 11, wherein thefirst sensing device comprises a digital camera.
 13. A non-transitorycomputer-readable medium or media storing computer-executableinstructions for directing a computer to perform a method forcontrolling the contents of a computer-generated image in conjunctionwith an enclosure that contains a plurality of sensors and that containsa projection device, the plurality of sensors comprising anaccelerometer and a gyro and a magnetometer and an image capturingdevice and a sensing device that is sensitive to a position of theenclosure and orientation of the enclosure; the method comprising thesteps of: causing the projection device to project a plurality ofdistinct points, causing the image capturing device to capture a firstimage, receiving data that is dependent on an output of at least one ofthe plurality of sensors, and using the data to control the contents ofthe computer-generated image when the data depends on at least a part ofthe first image and the first image contains a second image of at leastone of the plurality of distinct points.
 14. The non-transitorycomputer-readable medium or media according to claim 13, wherein each ofthe plurality of distinct points is projected substantiallysimultaneously with the plurality of distinct points.
 15. Thenon-transitory computer-readable medium or media according to claim 13,wherein each of the plurality of distinct points is projected in adirection that has a non-zero angle relative to the optical axis of theimage capturing device.
 16. The non-transitory computer-readable mediumor media according to claim 15, wherein the projection device comprisesa laser.
 17. The non-transitory computer-readable medium or mediaaccording to claim 16, wherein the laser is for projecting infraredlight, and the enclosure contains a computer-readable identificationcode that distinguishes the enclosure among a plurality of likeenclosures.
 18. The non-transitory computer-readable medium or mediaaccording to claim 13, wherein the enclosure contains acomputer-readable identification code that distinguishes the enclosureamong a plurality of like enclosures, and the plurality of sensorscomprises a pressure sensor.
 19. The non-transitory computer-readablemedium or media according to claim 13, wherein the sensing device issensitive to two independent positional coordinates of the enclosure andthree independent orientational coordinates of the enclosure.
 20. Thenon-transitory computer-readable medium or media according to claim 13,wherein the sensing device is sensitive to three independent positionalcoordinates of the enclosure and three independent orientationalcoordinates of the enclosure.